/var/log/rant

More Gripes and Misunderstandings about Dancer

2012-01-18T14:41:00.000-05:00

It has been suggested that I'm not using MVC separation, which is fair, as I don't really see MVC separation. I found the POD for Dancer::Plugin::Database, and have modified my package to use it. weather is the old one, weather2 is the new one using the plugin.

package TestApp ;
use Dancer ':syntax' ;
use Dancer::Plugin::Database ;
use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;
use DateTime ;
use Data::Dumper ;

our $VERSION = '0.1' ;

get '/' => sub {
    template 'index' ;
    } ;

get '/weather' => sub {
    my $dbh = db_connect() ;
    my $sql = <<SQL;
    # Last Eight Hours
    SELECT  time , AVG( temp_f )
    FROM weather WHERE zip = "47909"
    AND HOUR( TIMEDIFF( SYSDATE() , time ) ) < 9
    GROUP BY HOUR(time)
    ORDER BY time
SQL
    my $hr = $dbh->selectall_arrayref( $sql ) or croak $dbh->errstr ;
    my @hr ;
    for my $point ( @$hr ) {
        my ( $time, $temp ) = @$point ;
        my @time = split /\D+/ ,  $time ;
        my $dt = DateTime->new(
            year  => $time[0] ,
            month => $time[1] ,
            day   => $time[2] ,
            hour => $time[3] ,
            minute => $time[4] ,
            second => $time[5] ,
            ) ;
        my $data ;
        $data->{ time } = $dt->format_cldr( 'h:mm a' ) ;
        $data->{ temp } = sprintf '%.1f', $temp ;
        next if $data->{ time } !~ /00/ ;
        push @hr, $data ;
        }
    my $string = 'Temperature in West Lafayette for the last 8 hours' ;
    my $hr_ptr = \@hr ;
    template 'weather' , {
        string => $string ,
        weather => $hr_ptr
        } ;
    } ;

get '/weather2' => sub {
    my $string = 'Temperature in West Lafayette for the last 8 hours' ;
    my $sth = database->prepare(
        'SELECT  time , AVG( temp_f ) temp
        FROM weather WHERE zip = "47909"
        AND HOUR( TIMEDIFF( SYSDATE() , time ) ) < 9
        GROUP BY HOUR(time)
        ORDER BY time '
        ) ;
    $sth->execute( ) ;
    template 'weather' , {
        string => $string ,
        xxx => $sth->fetchall_arrayref( ) ,
        } ;
    } ;

true ;

I have been pointed to http://search.cpan.org/~xsawyerx/Dancer-1.3091/lib/Dancer/Tutorial.pod, and, beyond my wanting to be more specific in the munging of the data, weather is very similar to "Our First Route Handler". (How I store some data is not necessarily how I want to display some data.)

weather2 uses http://search.cpan.org/~bigpresh/Dancer-Plugin-Database-1.60/lib/Dancer/Plugin/Database.pm and you can see that there's much less code in there. The stuff I put into MyDB exists in Dancer::Plugin::Database, and I probably could mess with the data before sending it on, instead of hooking up $sth->fetchall_arrayref() to the output.

The big thing I see here, which I could do something about with weather but probably not with the plugin-using weather2, is the SQL code being here. I would much rather shove that SQL off into a library somewhere, which I've done elsewhere, than have it at this point. But maybe, that's an aspect of me not really getting the MVC thing yet.

I Can't Help Feeling Stupid Standing 'Round

2012-01-18T11:17:00.001-05:00

We're jumping from straight Perl and CGI.pm for web applications to an MVC, a jump we've been wanting to do for years. I've tried Catalyst a couple times, and both times, I spend most of my learning curve thinking "If I wrote this up without an MVC, I'd be done and on to the next thing by now", and in general, I get told much the same thing by my bosses, in the form of "move on to this other, non-MVC-related task".

But that was Catalyst, which seemingly brings along half of CPAN with it. This time, I'm trying Dancer with Template Toolkit, and I'm getting some progress. Here's a chunk of code I have working.

package TestApp ;
use Dancer ':syntax' ;
use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;
use DateTime ;

our $VERSION = '0.1' ;

get '/' => sub {
    template 'index' ;
    } ;

get '/weather' => sub {
    my $dbh = db_connect() ;
    my $sql = <<SQL;
    # Last Eight Hours
    SELECT  time , AVG( temp_f )
    FROM weather WHERE zip = "47909"
    AND HOUR( TIMEDIFF( SYSDATE() , time ) ) < 9
    GROUP BY HOUR(time)
    ORDER BY time
SQL
    my $hr = $dbh->selectall_arrayref( $sql ) or croak $dbh->errstr ;
    my @hr ;
    for my $point ( @$hr ) {
        my ( $time, $temp ) = @$point ;
        my @time = split /\D+/ ,  $time ;
        my $dt = DateTime->new(
            year  => $time[0] ,
            month => $time[1] ,
            day   => $time[2] ,
            hour => $time[3] ,
            minute => $time[4] ,
            second => $time[5] ,
            ) ;
        my $data ;
        $data->{ time } = $dt->format_cldr( 'h:mm a' ) ;
        $data->{ temp } = sprintf '%.1f', $temp ;
        next if $data->{ time } !~ /00/ ;
        push @hr, $data ;
        }
    my $vars = {
        string => 'Temperature in West Lafayette for the last 8 hours' ,
        weather => \@hr ,
        } ;

    template 'weather' , { var => $vars } ;
    } ;

true ;

A few notes here: MyDB.pm is a module I use to get a DBI object without putting the DB connection info into the code, making it safer for me to dump it in public like this. I've been collecting weather data for the West Lafayette area so I have a significant set to use when I want to play around with SQL and R.

Normally, when I pass things around, I pass, for example, a hash of hashes, then sort on the keys of the hash. Here, I'm making an array of hashes, so it's pre-sorted. I'm liking that. Still wrapping my head around how to put data where Template Toolkit can get to it. I like having my HTML as HTML, with occasional <code>[% FOREACH object IN array %] [% object.value %] [% END %] </code> sorts of thing added. I used to have lots of <code> print qq{<div> $div_contents </div>} ;</code> in my code, before I got here, where the coding standard is <code> print $cgi->div( $div_contents ) ; </code>, which is fine and all, but I always was very proud of my HTML and it feels good to be getting back into writing it in the midst of the coming of HTML5 and all.

But I'm curious if I did this the correct way of putting together the data, or if there's an easier better way that isn't apparent from the Dancer::Cookbook POD. It seems like I should be expecting a greater leap than this seems to give. What am I missing, if anything?

Ubuntu on the Big Screen

2012-01-14T01:20:00.000-05:00

So, Google TV, Apple TV, Roku, Boxee, and now Ubuntu TV. I think there's going to be problems making it a standard install because there's no standard setup on TVs. A big hamstring is the lack of Blu-Ray support in Linux (true last time I checked, maybe not now) but I can easily see a point soon where that doesn't matter anymore.

So, the set-top is the new desktop? And is this reason to go to Linux Mint or even straight Debian on machines you want to run at work?

Belkin Offers Wireless Home Automation

2012-01-11T11:06:00.000-05:00

Home Automation is cool. This has been the opinion of this blog since the beginning. I have tested some X10 stuff before, and have found data-over-power-line to be unreliable. This is why the wireless setup that was mentioned in Google I/O last year (but not in any way delivered, as far as I can tell ) draws so much interest from me, and why I feel the need to show this video here. Wireless is so the way to go with this stuff. I wonder if you can put together little command-line apps to turn things on and off, which you can then set up via crontab on your Linux box to turn the lights on or start the coffee machine at 7am or so.

I also like the motion sensor. In rooms where you're not going to want to darken while you're in them (bedroom, living room for movie night) and rooms where you're not likely going to be blocked from the motion sensor during your time there (bathroom, kitchen), lights on a motion sensor make all the sense in the world to me. I used to have one in my den and I loved it.

Belkin's normally the "Hey! Cheap cable! Cool!" company for me, but I'd be more than happy to give this one a shot.

"Did the HTTP Headers tell you about Comcast?"

2012-01-11T10:40:00.002-05:00

There's some buzz about an article in Forbes called "Why Best Buy is Going out of Business ... Gradually" (and the followup) about how specific decisions (such as training their workforce to talk to people about buying other stuff, not knowing what's in the store) is hurting their customer relations. Long article, go read.

Part of it is telling the story of a trip to Best Buy, looking for How To Train Your Dragon in 3D, when a sales person with no knowledge of Blu-Ray placement came up and tried to push him onto another cable system. I've come to expect that when I walk into Best Buy. It doesn't deter me from shopping at Best Buy yet, the way that the old-school, "ask for a page of personal information for a $2 cash purchase" kept me from shopping at Radio Shack for over 15 years, but it is annoying.

I went to Menards yesterday. I had three things I was looking for: stuff I saw on Make or Lifehacker or something where you dip the handles of your tools to get a comfortable and insulated grip (called Plasti Dip and to be found in the the same aisle as the Leathermen), cheap masking tape and Dremel tools I could use to recess the bottom of my son's Pinewood Derby car so the weights don't drag on the track. What I did not want is a young woman to try to sell me on Comcast. In my apartment complex, I have the choice between DishTV and cable-cutting, anyway, so I'm not the market anyway.

First guy I talked to didn't know about Plasti Dip, so he called a co-worker who directed me right to it. I don't count that as a customer service fail at all. What I do count as a fail is the free-range Comcast girl interrupting my tool-hunting experience, for two reasons. First, when men shop, they are expected to know what they want and where it is, or at least use the game-seeking sections of his brain to scope out the environment and find it. It's a hunt. There are people there whose job it is to maintain the game preserve and point out where to find the beast. The shopper respects the worker by interrupting him only after looking for himself first, and the worker respects the shopper by being available to ask questions but not forcing an interaction before the hunt has finished. That is the male way of shopping, and the Comcast girl breaks that.

Second, cable TV connects to home electronics via a coaxial connection in the back of the unit. It's more than a natural fit, it's an engineered fit. You're looking at big HD TVs? You'll want something to watch on it, and man, I have just the thing. Cable TV is as much a fit for hardware stores as Legos or tampons: not much.

I first heard of something like this in college: at the bar, a cute girl in a tight Jagermeister T-shirt comes up and asks if you've ever tried Jager, which you try because, hey, cute girl. Maybe there's even a Jager Face contest and she has a Polaroid camera (OK, think back to 1991 on this one, OK?) Suddenly, everyone you know drinks Jager shots and beer. I guess I don't mind it in the bar, in part because I don't do that anymore, but I increasingly mind it in the stores. I hope this is a dying trend.

My Song of Love for Unix

2012-01-06T12:30:00.002-05:00

My first computer was a DOS-running TRS-80 in the 1980s. In 1988, when I was heading off to college, my parents got me a Tandy PC with an 8088 processor and a green monochrome monitor that took a composite video cable. I still have it but I don't use it. The only modification I did on it was to add a modem (2400 baud, I think, but maybe 1200 baud) and I mostly used it to connect to BBSes and mostly to the campus' big machine, MUSIC (Multi-User System for Interactive Computing). I know I touched some GNU tools for DOS at the time, and I heard about UNIX systems and felt an awe about them that I don't think it's possible to express these days, when you can download Linux for free and all of Apple's computers run BSD, but I never got close to a real UNIX system back then.

In fact, the first I really touched a Unix system was when I went to school again, to get my Computer Science degree. I learned C as a subset of C++ on SunOS systems, and I have to say that I was somewhat underwhelmed with Unix. X looked positively crufty compared to Windows 95, much less MacOS, even in 1996. What I didn't get, what it took me a while to get, is that the windows don't really matter. Once you get into the Unix mindset, CDE is KDE is Gnome is Windows is MacOS. The way windowing doesn't matter, and what does matter is the command line.

And why is that? Windows has CMD and now Powershell, and Macs could be coerced to give you a prompt even back then, even if few people used them. What makes Unix systems different? In a word, pipes.

Pipes?

So, you have an MP3 of a song, somewhere in your music collection. For example, "Never Ending Love" by Delaney and Bonnie. You don't know where it is and you want to copy it to your phone. So, first you have to go through your music collection.

find Music/.

That gives you a big list of all your music. tl;dr

find Music/. | grep -i delaney

OK, so, that gives you the whole album, plus "Delaney's Dream" by Seth Austin, too.

find Music/. | grep -i delaney | grep -i "never ending"

And, if your music collection is mine, that cuts it down to two. And a great many other problems can be broken down in similar ways.

It gets deeper, but in essence, the command line, the basic way you connect with a machine, is a programming language in and of itself. You write small things that filter the information, learn the flags to control the output of the tools, and only display it when it is down to what you need to see. To the extent it is possible to script a windowing system, it is a cast-iron PITA. When you want to script things in the command line, it is baked in at a fundamental level.

So, if someone suggests you learn Unix, yeah, it's great to run Ubuntu and get the Unity shell (the current term for window manager, I guess), but you'll either be using things that are ported to all the platforms (Chrome, Firefox, Komodo Edit, in my case), things that are equivalent to things on other platforms (Nautilus vs Windows Explorer, for example) or things that are hoped to be equivalent to things on other platforms but just aren't (GIMP vs Photoshop). This makes Mac vs PC vs Linux largely a matter of taste, but the command line, or shell, is the thing that makes Unix different, that makes it powerful, which makes it powerful and user-friendly to those who it excepts as users.

Yes, I have just largely restated Neal Stephenson's "In the Beginning was the Command Line". I still enjoyed the trip, and hope you did too.

Information Diet Commentary, or Sniping At An Idea I Have Some Sympathy For

2012-01-01T15:39:00.001-05:00

I hit an idea on my Google+ feed about Information Diet. This has been presented as a summation of the concept:

#1 Productivity Tip: Spend 10% of your time consuming and 90% of your time producing. Make more stuff. Watch less. Read less. Do.

Really, this is hard to argue with, but hard to follow, as Tim O'Reilly said.

So, I go around looking at more. While I think the author of Information Diet and I probably wildly disagree on key points of ideology, I would guess that I could find lots of points of commonality. I'll have to do some reading to decide that.

But I found something else that I strongly and intensely disagree with.

#2 Decentralise your media consumption This is a bit of a heretical idea too - technology is geared towards central entertainment systems for our homes. But you know what? They are hateful, terrible devices. Ten years ago, if I'd wanted to listen to music, my flow chart would have looked like this:
SWITCH ON STEREO > INSERT CD > PRESS PLAY
Now it looks something like this
SWITCH ON PC > WAIT TO BOOT > LOG IN > MORE BOOTING > UPDATES START ROLLING > CLICK ON ITUNES > WAIT SEVERAL MINUTES WHILE IT BOOTS > GET BORED, OPEN EMAIL/BROWSER > ITUNES UPDATE REQUIRED > START DOWNLOAD > START CLICKING ON FEEDS > ITUNES UPDATE DOWNLOADED > INSTALLING > RESTARTING ITUNES, MORE WAITING > ITUNES OPEN > ! WINDOWS UPDATE FINISHED, MANDATORY REBOOT...

Funny, in 2001 and before, my process for listening to music, more often than not, was:

OPEN WINAMP > CHOOSE A SONG > PRESS PLAY

I was and still am a computer guy at that time, and because I was coding and otherwise working on the PC, my stuff is by and large already updated to current versions. If the "now" case from Sciencepunk is every single time, iTunes must suck. But yeah, for the last 10 years, my CDs would get ripped and filed almost immediately, and I would listen to the original media rarely.Let me contrast it with my listening to a CD in 2001:

 EITHER GO TO MY WALL OF CDS (HOME) OR MY SMALL MOBILE COLLECTION > LOOK FOR CD > LOOK AGAIN BECAUSE WITH THAT MANY CDS, YOU NEVER FIND IT ON FIRST ATTEMPT > IF FOUND, INSERT CD AND PRESS PLAY, ELSE FAIL

And things like Google Music and Spotify make this much much easier. And never ends in fail.

In Praise of Night-Hacking, or at least in hopes to understand

2011-12-16T23:21:00.001-05:00

Photo credit: pantulis

I code every day. It's what I do, and I'm thankful. But there is a problem with coding from 9am to 5pm.

Other people.

Yeah, you're coding for their benefit, and it is well and good and helpful (and secretly, an ego boost at times when you otherwise feel humbled by the tasks in front of you) to serve as tech support for other people, even when and sometimes especially when the task seems otherwise mundane to you.

But, when you have all that state in your head, the last thing you want to do is fix someone else's thing. It has had me nearly to tears before. When there's nobody else around, you don't have to worry about the house of cards you have in your head. That's when you can relax and get into it, man. You know, like a coding machine?

But that's not it. Or at least not all of it.

At least what a geek with a hat has to say. In essence, when you're tired, parts of your brain shut down, and those can be the "squirrel!" aspects that get in the way of the deep concentration you need to build, understand and modify the house of data cards in your head. And, with your focus and the lights of the computer screen, you can keep going until you drop off, and when you do, it's the deep, relaxing sleep that exhaustion brings, not the light, brittle sleep sometimes broken by insomnia you get when you're sleeping because it's time. (I once heard a professor opine that humans were really built for 25-hour days. I do believe that some.)

A big problem is, of course, that if your life doesn't live on hacker's hours, parts of your life fall away when you start living on hackers hours. Something to sleep on, so to speak.

Can You Identify The Bug?

2011-12-15T09:55:00.003-05:00

#!/usr/bin/perl
# 2g_run_reversal.pl

use 5.010 ;
use strict ;
use warnings ;
use Carp ;
use Data::Dumper ;
use Getopt::Long ;
use Pod::Usage ;

use lib '/home/ltl/lib' ;
use SecGenTools::Run 'get_run_data' ;
use SecGenTools::Accession 'get_accession_data' ;
use SecGenTools::Request 'get_request_data' ;

my $config      = config() ;
my $run_stub    = get_run_data( $config->{ run_id } ) ;
my $run         = $run_stub->{ $config->{ run_id } } ;
my $run_samples = get_run_sample( $config->{ run_id } ) ;
my %conversion =
    map { $_, $run->{ max_regions } + 1 - $_ } 1 .. $run->{ max_regions } ;

for my $region ( 1 .. $run->{ max_regions } ) {
   my $conversion = $conversion{ $region } ;
   for my $sample_id ( sort keys %$run_samples ) {
       my $sample = $run_samples->{ $sample_id } ;
       next if $sample->{ region } != $region ;
       $sample->{ region } = $conversion ;
       update_run_sample( $sample ) ;
       say join "\n\t" , $sample_id ,
           $sample->{ accession_id } ,
           $region ,
           $conversion ,
           ;
       }
   }

exit 1 ;

This is the core of the code. config() puts all the configuration options into a hash and returns it. get_run_data(), get_request_data(), andget_accession_data(), are wrappers around database calls.

An accession is an individual sample to be analyzed. A request is one or more accessions. A run is one or more requests, assigning the accessions to one of the regions. And, because this is a mapping between a data structure and a physical object, it is possible for the user to start at the wrong end. This program is supposed to take all the requests and reverse the regions. That is, for an 8-region run, all the accessions in region 1 should be in region 8 and vice versa.

This code puts everything in regions 1-4. It took me a morning to figure out why, although part of that was restoring the database to the way it was before.

It took me a while to get it, so stop here if you're still looking.

In a word, persistence.

$sample is a reference to the data in $run_samples. In my mind, my $sample = $run_samples->{ $sample_id } made a local copy, but it didn't. That meant that, when the code hits region 5, it works on the accessions just moved over from region 4, too, and so on.

And, in working around that, I decided that looping on region was useless except to massage the retentive control freak in me, and once I convinced myself of that, things got better. So, like the Monster at the End of the Book, the bug is me.

Lots of Small Rants

2011-12-07T00:15:00.001-05:00

I got three things on Black Friday, and the first one shouldn't even count. It's the e-book version of Programming Android from O'Reilly. Time was, when I had a technical question, O'Reilly books were the first place I'd look, but now, my first checks are Google and StackOverflow, so this is a bit of a nostalgia move, but there's enough of a blank spot in my mind to justify it. Now, just to spend some time and generate something.

The other two are cheap pieces of USB hardware from Newegg. The first is a USB Wifi dongle (shown right), or, as it reads, "802.11b b/g/n Wireless Adatper". I have to say, I like it better so far than the Netgear that has been going back and forth between a few Windows boxes I have. Plus, I like the form. It's nice and small.

The other is also an inexpensive piece of Rosewill kit, a USB dongle connecting to a remote to Windows Media Center. Right now, it's nonsensically connected to the laptop I'm writing this on, but I could easily see myself liking this. I have a wireless Logitech keyboard with gone-AWOL mouse and don't do much with whatever PC I have in the bedroom for lack of pointer, and while I have been using Boxee for bedroom media PC duties, I could see myself accepting Windows Media Center due to this one. Problem is, the PC I needed the USB dongle for is the the same one I needed the remote for, and because it's been repurposed from HTPC to desktop, as long as I need the WiFi dongle, I do not need the remote and vice versa.

I've been doing more with git, as previously mentioned today, and I think have hit the distinction for bare repositories. This has been kicking to most of the day, and it's a bit of duh I just had to work through to get this into my head. I've been making non-bare repositories and wanting them to behave like bare repositories, then tearing my hair out. I'll have to work out aliases or scripts for this stuff.

Book Review - Version Control by Example, or "Thank you, @eric_sink"

2011-12-06T11:34:00.001-05:00

For too long, I've worked in environments without version control. There's been backups, either real or virtual, and for the web part there's the Google cache (which saved by butt once as the webmaster for my LUG) but version control has always been something that I know I should be doing, but we've never done. (The one exception was doing temp work at the car parts company. There, we used code reviews and Synergy. Not the good Synergy, which gives you the ability use one keyboard and mouse across several computers in software, but the bad one, the ancient, slow and ponderous configuration management system, which is similar enough to version control that I cannot express the distinction.

So, here's something where I know I should be doing it, but I don't really know how to do it. I've heard enough about Git and have signed up with Github, which put it before Subversion and CVS and the rest of the choices. But, unfortunately, I hadn't heard enough about Git to really know what the heck I should be doing with it.

Then I saw a post by my internet friend Funnel Fiasco about Version Control by Example by Eric Sink, saying how good it was. And, as it turns out, Eric is much more about teaching people about how to not do stupid things than he is about sales, so you can download digital copies, browse through online, and he'll even send you a copy free.

I've had it in my bag for several months without cracking it open, which was stupid of me. Right now, I'm in a decent place at work, with little sitting with a tight deadline, so I have time to go through the book and start actually learning from it, and wow! I'm getting it! It makes sense. The core of the book is the same examples expressed in Subversion, Mercurial, Veracity and Git, so it serves as a Rosetta Stone, too, so if you know one, you have a step toward using the others if that's what your workplace or project uses.

Thank you, Eric Sink!

Speaking of git

2011-12-05T13:13:00.001-05:00

mount.pl, my script for automating the mounting and unmounting of remote file systems via sshfs, is now on github.

Questions about Git

2011-12-05T11:20:00.001-05:00

I'm reading Eric Sink's Version Control By Example, starting to hit the examples, and I'm finding that to be a bit of a problem.

Eric's example is one dev in Birmingham, UK and one in Birmingham, AL, writing C code and committing to a server in Cleveland. Right now, the dev team for my office are two coders separated by all of five feet, writing Perl code for one of two servers, one being the web server. I'm easing in to git, as you might guess by the reading material, and like most working environments I've been in, our version control system has been "copy a backup before you muck with the file", which I know is dumb and useless (especially in the lab, where we rely on RAID for data protection and thus don't really have backups). I'm kinda taking the lead on this, having been burned enough to want to protect myself, but I don't really know much of it.

I've been using git so far to keep track of changes locally, just doing git init and git commit within the local file system. Is this enough? Or do we really, truly need a server?

And the code sits in /home/user/web/cgi-bin/foo/bar/blee/quuz.cgi or /home/user/bin/hoge.pl, I'm wondering: should I have those be the directories to run git in? Or should I do the work in /home/varlogrant/dev/hoge.pl/ and copy hoge.pl over to /home/usr/bin when I'm happy with it and it has been committed?

I have a bigger question on how to take a large selection of interconnected Perl modules and make them 1) test driven in the real, chromatic-approved way, 2) working with git in a useful way, and 3) usable from ~/lib on several dissimilar systems. I have a cheap hack on #3, but if I break apart and reassemble the modules, I can probably do it cleaner and smarter.

'cat' considered not useless

2011-12-02T11:54:00.001-05:00

Consider this:

cat /path/to/my/file.log |
grep filter |
sed 'something, I don't really use sed a lot' |
grep filter_again |
lp

Many would write this off as "useless use of cat". I don't think so. I mean, functionally, sure. But it isn't all functional.

cat /path/to/my/file.log | # read data from file
grep filter |              # run a filter on file contents
sed 'something, I don't really use sed a lot' | #change file contents
grep filter_again |        # run another filter on contents
lp                         # send to printer

Compare to this:

grep filter cat |          # read data from file AND run a filter on file contents
sed 'something, I don't really use sed a lot' | #change file contents
grep filter_again |        # run another filter on contents
lp                         # send to printer

You have now overloaded grep and if you want to remove it, you now have more editing than reading a line.

Friend of the blog Patrick says that this use of cat is just as useless as comments. I tend to agree.

You Do Not Own Your Phone

2011-12-01T13:33:00.001-05:00

That is, unless you root it. Thank you Trevor Eckhart, and no thank you Carrier IQ.And Forbes reports that this could mean millions of violations of wiretap law.

Improving my SSHFS script

2011-11-28T09:46:00.001-05:00

I can name more than a dozen machines whose file systems I would want to mount from work. I don't mount those file systems all the time, but I do often enough that I have a Perl script I use to manage my SSHFS mounting and unmounting of them.

Here's a problem. This weekend, the servers for work were taken down for scheduled maintenance. That's fine, but it does mean that I had to remount a bunch of them this AM, and when the script went through the already-mounted filesystems, it would ask for a password and say "this is already mounted". What I probably want is some way of knowing quickly whether there's something mounted and then doing that check before I do the mount.

And this is what initially draws my attention: mount points are directories. And directories can contain files. It is a known trick for hackers to unmount shares, hide files in the mount point, and remount, where sysadmins won't notice it. It strikes me that I can touch a file like .unmounted into each mount point, and then look for that file on each remounting, and skip it if I can't -f that file.

But really, this has to be a solved problem somehow, so there might be a better way. Pointers?

ETA: The code, a few revisions back, is part of a previous post. A twitter response is leading me to start the process of finally using my github account and setting it up there. Will add again when I get to there.

I've Been Doin' Some Hard-Travelin', I Thought You Knowed

2011-11-16T13:02:00.001-05:00

Back to the Traveling Salesman.

What I had before was 11298 miles, using the shortest available path to an unconnected state capital, and it had problems, problems where the path already chosen forced a great amount of backtracking. Knots, my friend Mark calls them. The knots are the thing that looked really wrong to me.

So, I added another step.

I modified choose_shortest_path() so that it returned an array with the path. I then do some substitutions. Take two capitals, switch their order, and if that gets us shorter, go with that. Not randomly. Iteratively. First those next to each other, then those separated by one, then by two, up to 5. Then again. Five times.

This gets me to 10886 miles. So far. I'm doing it again, five times going from one to forty, just to see if we can get better than that, because the Washington-to-Arizona knot looks wrong to me, but that's a gut feeling, not a proven issue. That is a near-1100-mile leap, but using it seems to save me 412 miles, so it must work.

A CS professor once described NP-Complete problems as a license to hack, because there isn't an established best solution, you can play with it. This is a bit what I'm doing here. Certainly, this won't help you pack you knapsack, but if it helps you visit all the capitals that much faster, I'm happy.

#!/usr/bin/perl

# naive shortest-path determination - A little better

use 5.010 ;
use strict ;
use warnings ;
use Data::Dumper ;
use DBI ;

use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;

my $states    = get_states() ;
my $combos    = get_combos() ;
my $distances = get_distances() ;
my %shortest ;

#for my $start ( 1..48 ) {
#    my $state = $states->{ $start }->{ state } ;
#    my @path = choose_shortest_path( $start ) ;
#    my $dist = find_distance( @path ) ;
#    say join "\t", (sprintf '%02.2f' , $dist), $start, $state ;
#    }
#exit ;

my @path = choose_shortest_path( 23 ) ;
my $distance = find_distance( @path ) ;
say $distance ;
say as_google_url( separate_by_pipes( @path ) ) ;
say '' ;

my $path = \@path ;
for my $pass ( 1 .. 5 ) {
    for my $offset ( 1 .. 40 ) {
        my $start = 0 ;
        $path = massage_path( $start, $offset, $path ) ;
        my $distance = find_distance( @$path ) ;
        say join "\t", $pass , $offset, scalar @$path , $distance ;
        }
    }
say as_google_url( separate_by_pipes( @$path ) ) ;
say separate_by_pipes( @$path ) ;

exit ;

######## ######## ######## ######## ######## ######## ######## ########
sub choose_shortest_path {
    my @path = @_ ;
    return @path if scalar @path == 48 ;
    my $s_id    = shift @path ;
    my $state   = $states->{ $s_id }->{ state } ;
    my @choices = sort { #sort by distance
        $distances->{ $a }->{ distance } <=> $distances->{ $b }->{ distance }
        }
        grep { # haven't been chosen yet
                is_not_in_array( $combos->{ $_ }->{ state_id_1 }, \@path )
            and is_not_in_array( $combos->{ $_ }->{ state_id_2 }, \@path )
            }
        grep { # must have the state current state
               $combos->{ $_ }->{ state_id_1 } == $s_id
            or $combos->{ $_ }->{ state_id_2 } == $s_id
            } keys %$combos ;
    my $c     = shift @choices ; #shortest
    my $c_obj = $combos->{ $c } ;
    my ( $o ) = grep { $_ != $s_id } $c_obj->{ state_id_1 },
        $c_obj->{ state_id_2 } ;
    my $o_state = $states->{ $o }->{ state } ;
    my $d = $distances->{ $c }->{ distance } || 'x' ;
    return choose_shortest_path( $o, $s_id, @path ) ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub massage_path {
    my ( $a, $offset, $path ) = @_ ;
    my $b = $a + $offset ;
    my $alt ;
    @$alt = @$path ;
    if ( $b >= 48 ) { return $path ; }
    $alt->[ $a ] = $path->[ $b ] ;
    $alt->[ $b ] = $path->[ $a ] ;
    my $d1 = find_distance( @$path ) ;
    my $d2 = find_distance( @$alt ) ;
    $path = $alt if $d2 < $d1 ;
    return massage_path( $a + 1, $offset, $path ) ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub find_distance {
    my @path     = @_ ;
    my $distance = 0 ;
    for my $i ( 1 .. 47 ) {
        my ( $s1, $s2 ) = sort { $a <=> $b } $path[ $i ], $path[ $i - 1 ] ;
        my ( $combo ) = grep {
                   $combos->{ $_ }->{ state_id_1 } == $s1
                && $combos->{ $_ }->{ state_id_2 } == $s2
                }
            sort keys %$combos ;
        $distance += $distances->{ $combo }->{ distance } ;
        }
    return sprintf '%0.02f', $distance ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub is_not_in_array {
    my ( $num, $path ) = @_ ;
    for my $p ( @$path ) {
        return 0 if $num == $p ;
        }
    return 1 ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub get_states {
    my $dbh    = db_connect() ;
    my $sql    = 'SELECT * from state_capitals ORDER BY id' ;
    my $states = $dbh->selectall_hashref( $sql, 'id' ) or croak $dbh->errstr ;
    return $states ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub get_combos {
    my $dbh    = db_connect() ;
    my $sql    = 'SELECT * from combinations ORDER BY id' ;
    my $combos = $dbh->selectall_hashref( $sql, 'id' ) or croak $dbh->errstr ;
    return $combos ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub get_distances {
    my $dbh    = db_connect() ;
    my $sql    = 'SELECT * from distances ORDER BY id' ;
    my $combos = $dbh->selectall_hashref( $sql, 'id' ) or croak $dbh->errstr ;
    return $combos ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub separate_by_pipes {
    return join '|', @_ ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub as_mark_list {
    my ( $path ) = @_ ;
    return join '', map { $states->{ $_ }->{ st } }
        split m{\|}mx, $path ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub as_google_url {
    my ( $path ) = @_ ;
    my $url1 =
        'http://maps.google.com/maps/api/staticmap?path=color:0xff0000ff|weight:1|'
        ;
    my $url2 = '&size=500x400&sensor=false' ;
    my $body = join '|', map {
        join ',', $states->{ $_ }->{ latitude },
            $states->{ $_ }->{ longitude }
            }
        split m{\|}mx, $path ;
    return join '', $url1, $body, $url2 ;
    }

######## ######## ######## ######## ######## ######## ######## ########
sub key_from_value {
    my ( $v ) = @_ ;
    my %rev = reverse %shortest ;
    return $rev{ $v } ;
    }

Beyond Firebug to NYTProf

2011-11-10T09:40:00.001-05:00

Clearly, the problem is in the core application, not the CSS and JS surrounding it. And Firebug only covers the outside of the application.

So I got NYTProf going. A little search gave me the knowledge that calling a page at http://example.com/myprog.cgi?foo=bar is the same as calling it with perl myprog.cgi 'foo=bar', which is so good to know, especially since the addition of the NYTProf step is perl -d:NYTProf myprog.cgi 'foo=bar' .

So, I was able to shave off a second by caching. I could have HOP'd it and just used Memoize, but I like having all the details of a program visible so I don't get bit by something I can't see.

{
    # all lines of code with  %url_cache are new
    # URLs have been changed to protect the innocent.
    my %url_cache ;
    sub get_service_page {
        my ( $pi ) = @_ ;
        if ( $url_cache{ $pi } ) {
            return $url_cache{ $pi } ;
            }
        my $readfile = pi_Readfile() ;
        my $url = 'http://www.example.edu/~user/projects/XXXXX/' ;
        my $alt =
            'http://www.example.edu/~user/something_else.cgi'
            ;
        my $attr   = 'SGNAME_PUTATIVE' ;
        my $sgname = $readfile->{ $pi }->{ $attr } ;

        if ( ! defined $sgname || '' eq $sgname ) {
            $url_cache{ $pi } = $alt ;
            return $alt ;
            }
        $url =~ s/XXXXX/$sgname/ ;
        $url_cache{ $pi } = $url ;
        return $url ;
        }
    }

So, simply by holding onto that little piece of information instead of checking against the same pi_Readfile() each time, I was able to go from 2-3 seconds to 1.2 seconds. And, now that I'm seeing it, I could hold onto the data structure I get from pi_Readfile() the same way I hold onto the cache, and could probably tighten up even more.

I don't know why it didn't occur to me to do that in the first place....

Just Ran Firebug

2011-11-09T10:19:00.000-05:00

Pretty clear where the lag is, isn't it?

Even More Traveling, Even Less Sales

2011-11-07T13:38:00.001-05:00

Here's some table descriptions from MySQL, from which you should be able to reverse engineer the table creation. State Capitals


+-----------+-------------+------+-----+---------+----------------+
| Field     | Type        | Null | Key | Default | Extra          |
+-----------+-------------+------+-----+---------+----------------+
| id        | int(10)     | NO   | PRI | NULL    | auto_increment |
| state     | varchar(25) | YES  |     | NULL    |                |
| st        | varchar(2)  | YES  |     | NULL    |                |
| city      | varchar(25) | YES  |     | NULL    |                |
| latitude  | float(12,6) | YES  |     | NULL    |                |
| longitude | float(12,6) | YES  |     | NULL    |                |
+-----------+-------------+------+-----+---------+----------------+

Combinations - Connecting each one to each other


+------------+---------+------+-----+---------+----------------+
| Field      | Type    | Null | Key | Default | Extra          |
+------------+---------+------+-----+---------+----------------+
| id         | int(10) | NO   | PRI | NULL    | auto_increment |
| state_id_1 | int(10) | YES  |     | NULL    |                |
| state_id_2 | int(10) | YES  |     | NULL    |                |
+------------+---------+------+-----+---------+----------------+

Distances


+----------+-------------+------+-----+---------+----------------+
| Field    | Type        | Null | Key | Default | Extra          |
+----------+-------------+------+-----+---------+----------------+
| id       | int(10)     | NO   | PRI | NULL    | auto_increment |
| distance | float(12,6) | YES  |     | NULL    |                |
+----------+-------------+------+-----+---------+----------------+

I'll say again, I think I made a mistake by not including distance in the combination table. I didn't write perl code to put the state capital information into the database. I copied it from a source and recrafted it into SQL by hand.


INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 01 , "Delaware" , "DE" , "Dover" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 02 , "Pennsylvania" , "PA" , "Harrisburg" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 03 , "New Jersey, NJ" , "Trenton" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 04 , "Georgia" , "GA" , "Atlanta" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 05 , "Connecticut" , "CT" , "Hartford" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 06 , "Massachusetts" , "MA" , "Boston" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 07 , "Maryland" , "MD" , "Annapolis" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 08 , "South Carolina" , "SC" , "Columbia" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 09 , "New Hampshire" , "NH" , "Concord" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 10 , "Virginia" , "VA" , "Richmond" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 11 , "New York" , "NY" , "Albany" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 12 , "North Carolina" , "NC" , "Raleigh" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 13 , "Rhode Island" , "RI" , "Providence" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 14 , "Vermont" , "VT" , "Montpelier" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 15 , "Kentucky" , "KY" , "Frankfort" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 16 , "Tennessee" , "TN" , "Nashville" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 17 , "Ohio" , "OH" , "Columbus" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 18 , "Louisiana" , "LA" , "Baton Rouge" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 19 , "Indiana" , "IN" , "Indianapolis" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 20 , "Mississippi" , "MS" , "Jackson" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 21 , "Illinois" , "IL" , "Springfield" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 22 , "Alabama" , "AL" , "Montgomery" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 23 , "Maine" , "ME" , "Augusta" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 24 , "Missouri" , "MO" , "Jefferson City" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 25 , "Arkansas" , "AR" , "Little Rock" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 26 , "Michigan" , "MI" , "Lansing" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 27 , "Florida" , "FL" , "Tallahassee" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 28 , "Texas" , "TX" , "Austin" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 29 , "Iowa" , "IA" , "Des Moines" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 30 , "Wisconsin" , "WI" , "Madison" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 31 , "California" , "CA" , "Sacramento" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 32 , "Minnesota" , "MN" , "Saint Paul" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 33 , "Oregon" , "OR" , "Salem" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 34 , "Kansas" , "KS" , "Topeka" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 35 , "West Virginia" , "WV" , "Charleston" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 36 , "Nevada" , "NV" , "Carson City" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 37 , "Nebraska" , "NE" , "Lincoln" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 38 , "Colorado" , "CO" , "Denver" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 39 , "North Dakota" , "ND" , "Bismarck" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 40 , "South Dakota" , "SD" , "Pierre" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 41 , "Montana" , "MT" , "Helena" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 42 , "Washington" , "WA" , "Olympia" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 43 , "Idaho" , "ID" , "Boise" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 44 , "Wyoming" , "WY" , "Cheyenne" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 45 , "Utah" , "UT" , "Salt Lake City" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 46 , "Oklahoma" , "OK" , "Oklahoma City" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 47 , "New Mexico" , "NM" , "Santa Fe" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 48 , "Arizona" , "AZ" , "Phoenix" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 49 , "Alaska" , "AK" , "Juneau" ) ;
INSERT INTO state_capitals ( id , state , st , city ) VALUES ( 50 , "Hawaii" , "HI" , "Honolulu" ) ;

The latitudes and longitudes were also hand-crafted.


UPDATE state_capitals SET latitude="32.361538", longitude="-86.279118" where state = "Alabama" ;
UPDATE state_capitals SET latitude="58.301935", longitude="-134.419740" where state = "Alaska" ;
UPDATE state_capitals SET latitude="33.448457", longitude="-112.073844" where state = "Arizona" ;
UPDATE state_capitals SET latitude="34.736009", longitude="-92.331122" where state = "Arkansas" ;
UPDATE state_capitals SET latitude="38.555605", longitude="-121.468926" where state = "California" ;
UPDATE state_capitals SET latitude="39.7391667", longitude="-104.984167" where state = "Colorado" ;
UPDATE state_capitals SET latitude="41.767", longitude="-72.677" where state = "Connecticut" ;
UPDATE state_capitals SET latitude="39.161921", longitude="-75.526755" where state = "Delaware" ;
UPDATE state_capitals SET latitude="30.4518", longitude="-84.27277" where state = "Florida" ;
UPDATE state_capitals SET latitude="33.76", longitude="-84.39" where state = "Georgia" ;
UPDATE state_capitals SET latitude="21.30895", longitude="-157.826182" where state = "Hawaii" ;
UPDATE state_capitals SET latitude="43.613739", longitude="-116.237651" where state = "Idaho" ;
UPDATE state_capitals SET latitude="39.783250", longitude="-89.650373" where state = "Illinois" ;
UPDATE state_capitals SET latitude="39.790942", longitude="-86.147685" where state = "Indiana" ;
UPDATE state_capitals SET latitude="41.590939", longitude="-93.620866" where state = "Iowa" ;
UPDATE state_capitals SET latitude="39.04", longitude="-95.69" where state = "Kansas" ;
UPDATE state_capitals SET latitude="38.197274", longitude="-84.86311" where state = "Kentucky" ;
UPDATE state_capitals SET latitude="30.45809", longitude="-91.140229" where state = "Louisiana" ;
UPDATE state_capitals SET latitude="44.323535", longitude="-69.765261" where state = "Maine" ;
UPDATE state_capitals SET latitude="38.972945", longitude="-76.501157" where state = "Maryland" ;
UPDATE state_capitals SET latitude="42.2352", longitude="-71.0275" where state = "Massachusetts" ;
UPDATE state_capitals SET latitude="42.7335", longitude="-84.5467" where state = "Michigan" ;
UPDATE state_capitals SET latitude="44.95", longitude="-93.094" where state = "Minnesota" ;
UPDATE state_capitals SET latitude="32.320", longitude="-90.207" where state = "Mississippi" ;
UPDATE state_capitals SET latitude="38.572954", longitude="-92.189283" where state = "Missouri" ;
UPDATE state_capitals SET latitude="46.595805", longitude="-112.027031" where state = "Montana" ;
UPDATE state_capitals SET latitude="40.809868", longitude="-96.675345" where state = "Nebraska" ;
UPDATE state_capitals SET latitude="39.160949", longitude="-119.753877" where state = "Nevada" ;
UPDATE state_capitals SET latitude="43.220093", longitude="-71.549127" where state = "New Hampshire" ;
UPDATE state_capitals SET latitude="40.221741", longitude="-74.756138" where state = "New Jersey" ;
UPDATE state_capitals SET latitude="35.667231", longitude="-105.964575" where state = "New Mexico" ;
UPDATE state_capitals SET latitude="42.659829", longitude="-73.781339" where state = "New York" ;
UPDATE state_capitals SET latitude="35.771", longitude="-78.638" where state = "North Carolina" ;
UPDATE state_capitals SET latitude="48.813343", longitude="-100.779004" where state = "North Dakota" ;
UPDATE state_capitals SET latitude="39.962245", longitude="-83.000647" where state = "Ohio" ;
UPDATE state_capitals SET latitude="35.482309", longitude="-97.534994" where state = "Oklahoma" ;
UPDATE state_capitals SET latitude="44.931109", longitude="-123.029159" where state = "Oregon" ;
UPDATE state_capitals SET latitude="40.269789", longitude="-76.875613" where state = "Pennsylvania" ;
UPDATE state_capitals SET latitude="41.82355", longitude="-71.422132" where state = "Rhode Island" ;
UPDATE state_capitals SET latitude="34.000", longitude="-81.035" where state = "South Carolina" ;
UPDATE state_capitals SET latitude="44.367966", longitude="-100.336378" where state = "South Dakota" ;
UPDATE state_capitals SET latitude="36.165", longitude="-86.784" where state = "Tennessee" ;
UPDATE state_capitals SET latitude="30.266667", longitude="-97.75" where state = "Texas" ;
UPDATE state_capitals SET latitude="40.7547", longitude="-111.892622" where state = "Utah" ;
UPDATE state_capitals SET latitude="44.26639", longitude="-72.57194" where state = "Vermont" ;
UPDATE state_capitals SET latitude="37.54", longitude="-77.46" where state = "Virginia" ;
UPDATE state_capitals SET latitude="47.042418", longitude="-122.893077" where state = "Washington" ;
UPDATE state_capitals SET latitude="38.349497", longitude="-81.633294" where state = "West Virginia" ;
UPDATE state_capitals SET latitude="43.074722", longitude="-89.384444" where state = "Wisconsin" ;
UPDATE state_capitals SET latitude="41.145548", longitude="-104.802042" where state = "Wyoming" ;

The distances themselves were generated mathematically, with the help of Google and Wikipedia to find the how-to.


#!/usr/bin/perl

use 5.010 ;
use strict ;
use warnings ;
use Data::Dumper ;
use DBI ;

use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;

use subs qw{ get_combos get_states set_distance } ;

my $pi = atan2( 1, 1 ) * 4 ;
my $states = get_states() ;
my $combos = get_combos() ;

for my $combo ( (sort { $a<=>$b } keys %$combos ) ) {
    my $c_obj = $combos->{$combo} ;
    my ( $state_1 , $state_2 ) =  sort { $a <=> $b } $c_obj->{ state_id_1 } , $c_obj->{ state_id_2 } ;
    my $obj_s1 = $states->{ $state_1 } ;
    my $obj_s2 = $states->{ $state_2 } ;
    my $dist = haversine(
            $obj_s1->{ latitude } , $obj_s1->{ longitude } ,
            $obj_s2->{ latitude } , $obj_s2->{ longitude } ) ;
    say $combo ;
    say join ' - ' ,
    ( join ', ' , $obj_s1->{ city } , $obj_s1->{ state } ) ,
    ( join ', ' , $obj_s2->{ city } , $obj_s2->{ state } ) ;
    say join "\t" , '' , $dist . ' miles';
    set_distance( $combo , $dist ) ;
    }

sub get_states {
    my $dbh = db_connect() ;
    my $sql = 'SELECT * from state_capitals ORDER BY id' ;
    my $states = $dbh->selectall_hashref( $sql , 'id' ) or croak $dbh->errstr;
    return $states ;
    }
sub get_combos {
    my $dbh = db_connect() ;
    my $sql = 'SELECT * from combinations ORDER BY id' ;
    my $combos = $dbh->selectall_hashref( $sql , 'id' ) or croak $dbh->errstr;
    return $combos ;
    }
sub set_distance {
    my ( $combo , $dist ) = @_ ;
    my $dbh = db_connect() ;
    my $sql = "INSERT INTO distances ( id , distance ) VALUES ( $combo , $dist ) " ;
    say $sql ;
    $dbh->do( $sql ) or croak $dbh->errstr;
    }

sub haversine {
    my ( $lat1, $lon1, $lat2, $lon2 ) = @_ ;

    my $theta = $lon1 - $lon2 ;
    my $dist =
        sin( deg2rad( $lat1 ) ) *
        sin( deg2rad( $lat2 ) ) +
        cos( deg2rad( $lat1 ) ) *
        cos( deg2rad( $lat2 ) ) *
        cos( deg2rad( $theta ) ) ;

    $dist = acos( $dist ) ;
    $dist = rad2deg( $dist ) ;
    $dist = $dist * 60 * 1.1515 ;
    return sprintf '%5.2f' , $dist ;
    }

sub acos {
    my ( $rad ) = @_ ;
    my $ret = atan2( sqrt( 1 - $rad**2 ), $rad ) ;
    return $ret ;
    }

sub deg2rad {
    my ( $deg ) = @_ ;
    return ( $deg * $pi / 180 ) ;
    }

sub rad2deg {
    my ( $rad ) = @_ ;
    return ( $rad * 180 / $pi ) ;
    }

More Details on Traveling Salesman

2011-11-07T13:12:00.001-05:00

I have just received mail from a programmer in France who is interested in learning Perl and asked about some of the constructs in my Traveling Salesman code, specifically asking about MyDB.pm and do_connect. That module and that function are about one thing: getting a DBI object without having to have my login and password and all that within the body of the program, so I can do things like paste it into my blog without worrying. Adapted, it looks like this:

package MyDB ;
use strict;
use warnings;
use DBI;

use Exporter qw(import);
our %EXPORT_TAGS = ('all' => [ qw(
                                   db_connect
                            ) ],
                    );
our @EXPORT_OK   = ( @{$EXPORT_TAGS{'all'}} );
our $VERSION = 0.0.1;
our %_DB = (
    default => {
        user       => 'YouDontGetThis',
        password   => 'YouDontGetThis',
        host       => 'YouDontGetThis',
        port       => '3306',
        database   => 'YouDontGetThis',
        },
    test => {
        user       => 'YouDontGetThis',
        password   => 'YouDontGetThis',
        host       => 'YouDontGetThis',
        port       => '3306',
        database   => 'YouDontGetThis',
        },
    );

my $_db_params  = '';       # String of current database parameters.
my $_dbh;                   # Save the handle.

sub db_connect {
    my ($param_ptr, $attr_ptr) = @_;

    # If database is already opened then check for a fast return.

    if (defined $_dbh &&
        (!defined $param_ptr || $param_ptr eq ''))    { return $_dbh }

    # Check for a different set of parameters to use via a the name (string)
    #   of the parameter (e.g., 'test').

    my $which_db = 'default';

    if (defined $param_ptr && ref($param_ptr) eq '' && $param_ptr ne '') {
        if (defined $_DB{$param_ptr})   { $which_db = $param_ptr }
        else { return; }
    }

    # Get the base parameters ... copy and flatten from global array

    my %params = ();
    my %attr   = ();

    foreach (keys %{$_DB{$which_db}} ) {
        $params{$_} = $_DB{$which_db}{$_};
        }

    # Add in extra parameters if given and if the database is not the default.

    if (defined $param_ptr
        && ref($param_ptr) eq 'HASH'
        && (!defined $param_ptr->{database} ||
             $param_ptr->{database} ne 'default') ) {

        foreach (keys %{$_DB{default}})  {
            if (defined $param_ptr->{$_}) { $params{$_} = $param_ptr->{$_} }
            }
        }

    if (defined $attr_ptr && ref($attr_ptr) eq 'HASH') {
        foreach (keys %$attr_ptr)  { $attr{$_} = $attr_ptr->{$_} }
        }

    # Now make up an order string of the parameters so that we can compare
    #   them to the old ones.

    my $new_db_params = '';
    foreach (sort keys %params)  { $new_db_params .= $params{$_} };

    # Can also do a quick return if params are same as old ones

    if (defined $_dbh && $new_db_params eq $_db_params)  {
        return $_dbh;
        }

    # At this point either the database has never been opened or
    #   new parameters are to be used. Close database and reopen.

    $_db_params = $new_db_params;

    if (defined $_dbh) { $_dbh->disconnect }    # no error check

    my $source = "dbi:mysql:$params{database}:$params{host}:$params{port}";

    $_dbh = DBI->connect($source, $params{user},
                               $params{password}, \%attr);

    return $_dbh;

    } # End of db_connect

1;

For the particulars, I used something, I think WolframAlpha, to get the latitude and longitude of each capital, then looked into some geometry to calculate the distances. Perhaps I should include some database dumps here. for that info.

How I create my web applications

2011-11-07T12:35:00.002-05:00

I handle our web stuff from soup to nuts, so here's a little bit of my methodology on how I do that work.

I think about what we're supposed to do and what we're supposed to store, and try to express it in SQL, culminating in the creation of a table in our MySQL database.
I write generalized functions for CRUD (creation, reading, updating and deleting) as needed, and create or add to existing Perl modules. I also make testing scripts for these functions to run on the command line.
I write the read functions into a Perl-driven CGI program. I'm old-school enough that each attempt to learn a framework such as Catalyst leaves me frustrated. Full creation of an element is usually handled in this program.
I write a jQuery-based Javascript module to run within the CGI that allows me to collect and add to all the information I need to make modifications.
I write an AJAX backend program in Perl that passes JSON back and forth between the client and server.

Right now, I'm in that last step. I know there are a few things that I need to start doing. I need to have development, test and production streams going for all this stuff. I need to have much more git going on. And, often there are small tweaks on the CSS throughout this process.

Traveling Salesman without Farmer's Daughter

2011-11-03T10:47:00.001-04:00

Have posted some related details to the problem, specifically some database access boilerplate.

The challenge is not traditional Traveling Salesman, which brings you back to the start. In this case, this is all the state capitals in the continental US, and the challenge is to get through all of them in the shortest distance, but was not getting back to the start.

This is my naive solution, which is starting from what I judged to be the furthest east (Maine) and going westward, choosing the shortest capital-to-capital distance. This is easy, and it avoids the biggest pitfall, the one that makes this a named problem.

Starting at one capital, you have 47 choices. For each of them, there are then 46 choices each, and then 45, and so on. The notation for that is n! (as opposed to !n which means not n) and it is big. 1.24x10⁶¹. This means that generating all possible paths would take forever even on big iron. The CS term is nondeterministic polynomial, or NP. Traveling Salesman is NP-Complete, IIRC. What this means is that, while finding a relatively fast way through this is pretty easy, finding the provably shortest isn't.

But I'm sure I can do better than this naive solution. The backtracking to get Vermont in makes me think that starting with Vermont and New Hampshire might be the better solution, and the jump from Tennessee to Michigan tells me that simple shortest-path is not the best solution there. I can generate this fast with simple recursion, getting a simple ordered list, doing the transforms that could tweak this into a faster path is something I don't really know how to do, code-wise.

Here's my Perl code, including some dyked-out bits covering some other cases. The first thing I can think of is to check every edge and each time two edges cross, switch the order of the second node for each edge. I think that could work, if I could decide how to code it.

#!/usr/bin/perl

# naive shortest-path determination - sucks

use 5.010 ;
use strict ;
use warnings ;
use Data::Dumper ;
use DBI ;

use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;

my $states    = get_states() ;
my $combos    = get_combos() ;
my $distances = get_distances() ;
my %shortest ;

#for my $start ( 1..48 ) {
#    my $state = $states->{ $start }->{ state } ;
#    my $dist  = choose_shortest_path( $start ) ;
#    say join "\t", (sprintf '%02.2f' , $dist), $start, $state ;
#    }
#exit ;

# 23 = maine

#
##say 'long' ;
##say choose_longest_path( 23 ) ;
##say '' ;

#say 'short' ;
#choose_shortest_path( 23 ) ;
#say Dumper \%shortest ;
#say '' ;
#
#my ( $s ) = sort { $shortest{ $a } <=> $shortest{ $b } } keys %shortest ;
#
#say as_mark_list( $s ) ;
#say '' ;
#say as_google_url( $s ) ;
#say '' ;

sub as_mark_list {
    my ( $path ) = @_ ;
    return join '', map { $states->{ $_ }->{ st } }
        split m{\|}mx, $path ;
    }

sub as_google_url {
    my ( $path ) = @_ ;
    my $url1 =
        'http://maps.google.com/maps/api/staticmap?path=color:0xff0000ff|weight:1|'
        ;
    my $url2 = '&size=500x400&sensor=false' ;
    my $body = join '|', map {
        join ',', $states->{ $_ }->{ latitude },
            $states->{ $_ }->{ longitude }
            }
        split m{\|}mx, $path ;
    return join '', $url1, $body, $url2 ;
    }

sub key_from_value {
    my %rev = reverse %shortest ;
    my ( $v ) = @_ ;
    return $rev{ $v } ;
    }

sub choose_shortest_path {
    my @path = @_ ;
    do {
        #say join '|', reverse map {
            #join ',',
        #        $states->{ $_ }->{ latitude },
        #        $states->{ $_ }->{ longitude }
        #        } @path ;
        #say as_mark_list( join '|', @path ) ;
        #say as_google_url( join '|', @path ) ;
        return 0 ;
        }
        if scalar @path == 48 ;

    #say join ' ' , scalar @path , '-' , @path ;
    my $s_id    = shift @path ;
    my $state   = $states->{ $s_id }->{ state } ;
    my @choices = sort {
        $distances->{ $a }->{ distance } <=> $distances->{ $b }->{ distance }
        }
        grep {
                is_not_in_array( $combos->{ $_ }->{ state_id_1 }, \@path )
            and is_not_in_array( $combos->{ $_ }->{ state_id_2 }, \@path )
            }
        grep {
               $combos->{ $_ }->{ state_id_1 } == $s_id
            or $combos->{ $_ }->{ state_id_2 } == $s_id
            } keys %$combos ;
    my $c     = shift @choices ;
    my $c_obj = $combos->{ $c } ;
    my ( $o ) = grep { $_ != $s_id } $c_obj->{ state_id_1 },
        $c_obj->{ state_id_2 } ;
    my $o_state = $states->{ $o }->{ state } ;
    my $d = $distances->{ $c }->{ distance } || 'x' ;

    #say $state ;

    #say join "\t", '', $d, $c, $s_id, $state, $o, $o_state ;
    #say join "\t", '', join ' ' , @path ;
    #say '' ;
    my $dist = choose_shortest_path( $o, $s_id, @path ) ;
    return $dist + $d ;
    }

#sub choose_longest_path {
#    my @path = @_ ;
#    do {
#        say join '|', reverse map {
#            join ',',
#                $states->{ $_ }->{ latitude },
#                $states->{ $_ }->{ longitude }
#                } @path ;
#        say as_mark_list( join '|', @path ) ;
#        say as_google_url( join '|', @path ) ;
#        return 0 ;
#        }
#        if scalar @path == 48 ;
#
#    #say join ' ' , scalar @path , '-' , @path ;
#    my $s_id    = shift @path ;
#    my $state   = $states->{ $s_id }->{ state } ;
#    my @choices = sort {
#        $distances->{ $a }->{ distance } <=> $distances->{ $b }->{ distance }
#        }
#        grep {
#                is_not_in_array( $combos->{ $_ }->{ state_id_1 }, \@path )
#            and is_not_in_array( $combos->{ $_ }->{ state_id_2 }, \@path )
#            }
#        grep {
#               $combos->{ $_ }->{ state_id_1 } == $s_id
#            or $combos->{ $_ }->{ state_id_2 } == $s_id
#            } keys %$combos ;
#    my $c     = pop @choices ;
#    my $c_obj = $combos->{ $c } ;
#    my ( $o ) = grep { $_ != $s_id } $c_obj->{ state_id_1 },
#        $c_obj->{ state_id_2 } ;
#    my $o_state = $states->{ $o }->{ state } ;
#    my $d = $distances->{ $c }->{ distance } || 'x' ;
#
#    #say $state ;
#
#    #say join "\t", '', $d, $c, $s_id, $state, $o, $o_state ;
#    #say join "\t", '', join ' ' , @path ;
#    #say '' ;
#    my $dist = choose_longest_path( $o, $s_id, @path ) ;
#    return $dist + $d ;
#    }

sub is_not_in_array {
    my ( $num, $path ) = @_ ;
    for my $p ( @$path ) {
        return 0 if $num == $p ;
        }
    return 1 ;
    }

sub get_states {
    my $dbh    = db_connect() ;
    my $sql    = 'SELECT * from state_capitals ORDER BY id' ;
    my $states = $dbh->selectall_hashref( $sql, 'id' ) or croak $dbh->errstr ;
    return $states ;
    }

sub get_combos {
    my $dbh    = db_connect() ;
    my $sql    = 'SELECT * from combinations ORDER BY id' ;
    my $combos = $dbh->selectall_hashref( $sql, 'id' ) or croak $dbh->errstr ;
    return $combos ;
    }

sub get_distances {
    my $dbh    = db_connect() ;
    my $sql    = 'SELECT * from distances ORDER BY id' ;
    my $combos = $dbh->selectall_hashref( $sql, 'id' ) or croak $dbh->errstr ;
    return $combos ;
    }

Android Dreams

2011-10-25T15:58:00.000-04:00

android dreams from Samuel Cockedey on Vimeo.

Split Attention

2011-10-24T13:00:00.001-04:00

If anyone has a solution, please tell me.

I have some tasks. The one that's almost off my front burner is a conversion job. There's a gob of files that exist in the worst possible XML format, which is there's one entry in the XML, which is a huge compressed BLOB (binary large object) of unknown format. The files have a name that say what kind of job created it and what day, but if you don't know which project went through which day, it gives you no help. We can export as (a more real) XML, which gives us the sample information we need. Thing is, there are over 1500 samples to be converted, which really cannot be a by-hand operation.

So, I'm working with Sikuli again, this time on Windows not Ubuntu. I've done the best I can, but this is a program written in Jython (a Java [not my favorite] implementation of Python[not my favorite]) running in Sikuli (taking over the keyboard and mouse) going over Samba to grab the configuration and data files, running in 32-bit Windows 7 on a VirtualBox instance. There's lots and lots of not-necessarily-stable involved here, which means I have to keep checking on it to ensure it keeps running. Which means, while it is going, I can't really throw my mind at any task that really takes my attention, which, basically, means any task I need to do at work, which makes me feel terribly unproductive, even as I know that it's much more productive than if I was doing all that crud by hand.

So, until I can code up a way that's much more repeatable and stable, or at least until I get well past the 10% completion this project is at about now, I can't really focus elsewhere.

We Solved It! Return of the PAL Problem

2011-10-21T00:48:00.001-04:00

Remember what I was saying about the campus networking issue I've been discussing? Long story short: My phone connects to our PEAP and THAWTE protected campus network just fine, but once I am on, the network cannot do anything. But only with Android. Windows is happy with PAL2.0. iOS is happy with PAL2.0, and if I recall the last time I booted the poor, battery-starved old Linux laptop I have in my bookcase at work, desktop Linux is happy with PAL2.0.

But Android is not. Which is annoying.

I have of course went to the helpdesk, and that's where it gets a bit interesting. There's two groups I've hit: the helpdesk, who have found this problem across different Android versions, across different carriers, different handset manufacturers, without finding any common indicator, and the networking group, who, without more to go on, have entered the go away kid, you bother me zone, considering it a failure in user configuration.

I feel I should point out that, for the most part, my co-worker Rick keeps his Android phone all but off when he's in the office. I, on the other hand, tend to hook my phone into my Greater Audio System (Windows and another 1/8" cable [either laptop or phone, depending] run into a y-cable plugged into the audio-in of my Linux box, which I config to go direct to the audio out, because I don't have a mixer, and then into a speaker and to my headphones) so I can get my phone's notification beeps and podcasts along with the other audio I have at work. So I use the network and feel it when it isn't available. So, while I was pushed by usage, for Rick it was a question of curiosity. Which seems to be enough.

When you think about networking, if you think about networking, you probably think about your IP address akin to a phone number, which kinda works and kinda doesn't. Assume we have an address of 8.8.8.8 (which I don't: that's the IP address of Google's open DNS server, which I kinda like). I can directly connect to any machine on my subnet, which could easily be 8.8.8.7 and 8.8.8.15 or further out, and if I can't find what I want on my local network, the traffic goes out the gateway to the hierarchically higher network. (There can also be down, in addition to up, but that's not important right now.)

One way to look around the local network is by using MAC addresses and routing tables, but that's too low level for this discussion. We use subnet masks. It's a series of 1s and 0s, in that order, which is used to tell if IP address A is in the same subnet as IP address B. A common netmask would look like 11111111111111111111111100000000. Clearly, that's hard for people to deal with, so we would write that as 255.255.255.0. First, each of those four breaks out 8 spaces, which are binary representations of positive integers between 0 and 255. Here's a table of what the allowed numbers are:

00000000	0
10000000	128
11000000	192
11100000	224
11110000	240
11111000	248
11111100	252
11111110	254
11111111	255

So, if the subnet mask is, 255.255.0.0, that means the mask in binary is 11111111 11111111 00000000 00000000. The digit that share a space with 1 will be the same if the IP address is on the same subnet, and different if it's 0. Google's 8.8.8.8 would be 00001000 00001000 00001000 00001000, the neighbor 8.8.8.9 would be 00001000 00001000 00001000 00001001. Diff's at the end, in the zero space, and thus same subnet. 8.9.8.8 would be 00001000 00001001 00001000 00001000, and that diff would be in the ones, and thus different.

The subnet mask sent out by DHCP was 255.255.240.0. iOS and Windows are just peachy with that. Rick noticed that the IP addresses were a little higher than he would expect. He suggested that we enter in static IP addresses based upon what we got via DHCP (using a tool like ifconfig to tell you it all) but with 224 as the third octet, and it worked like a charm. Go us!

I don't like PAL2.0

2011-10-19T16:36:00.001-04:00

Purdue has a wireless network, called Purdue Air Link, or PAL2.0. In essence, you log on to the wireless the same way you'd log onto most other university computing equipment.

It works fine on my co-workers' iProducts.

It works fine on my laptop.

In most places, it works fine for me. And not just "at my home". Other places on campus, other places with PAL2.0.

But not in my office. In my office, I can connect -- phone says I have connection to network, phone says I have an IP address -- but cannot do anything with them. And the tools I would normally try to use to get a better idea of what's going on, such as ping, nslookup, traceroute and the like, are not on my phone. (Maybe I need to look into those.)

And, I should say, it affects my co-worker's Android phone, too.

I don't understand an issue that only occurs with Android phones, but it seems I have one. I think I'm getting an IP address, but I can't route.

Mad Mad Mad at Komodo Edit

2011-10-17T10:31:00.000-04:00

I program, and for most small jobs, and for most complex find/replace jobs, I prefer vi. I have tried EMACS, but I never really got it. But I've hit a point that, when I'm doing most of my coding, I prefer an editor that behaves a bit more like a word processor. I want a graphical editor. I have tried a few. The one I use these days is Komodo Edit from Activestate, which I use some on Windows but primarily on Unix.

I develop and maintain a few web apps for the lab, and that's everything from SQL to Perl libraries to Perl applications to HTML to Javascript. I'm often wanting to make sure that the page I'm writing is working right with the AJAX Javascript code I'm writing for it, which is connecting to the CGI Perl code I'm writing, and that it is working correctly with the modules. Since they're at the far ends of the user experience, the Perl module and the HTML are tabs in one editor window, with the client-side Javascript in an other window and the server side Perl in another. Both my journalism school page layout background and my history with 80x24 terminals has lead me to believe that you don't want to go too wide, so I try to keep my code within 75 to 80 characters per line. My main monitor is 1680x1050, and I can have my code at a readable font size and still have three editor windows in one monitor, leaving my other for terminals (I have four to six open post of the time) and my Windows monitor open to a web browser.

This means that I CANNOT have three open and not partially-hidden editor windows on one screen. This used to be no problem for Komodo Edit, and is a deal-breaker for me.

So, what other mature modern graphical text editors for Linux are out there?

Who is Dennis Ritchie and Why should I care?

2011-10-13T09:54:00.000-04:00

You can check Wikipedia for the fuller explanation, but here's the summary: He was a researcher at Bell Labs. While he was there, he worked on the Multics system, and as an offshoot, created the C language and the Unix operating system. The C language was not tightly tied to the specific system it came from, which meant that things written in it could reasonably be ported from one set of hardware to another.

He, Ken Thompson and Brian Kernighan are among the set of people whose creations made modern computation possible. I throw into that list Bob Metcalfe (ethernet) and Douglas Engelbart (windowing systems and the mouse). Others came along after and made things better, smaller, faster, more integrated and more beautiful, which is all very important, but Ritchie is among those who made things possible. He will be missed.

Dennis MacAlistair Ritchie 1941-2011

Open Computing, Taken To The Next Level

2011-10-12T01:50:00.002-04:00

Hat-tip MakerBlog

This is the Doma Pro PCI, a black acrylic "case" for a computer. I say "case" because, while there are enclosures for the optical and hard drive, it's not really a case because it doesn't close. All your electronic bits are open to the world.

Which is neat, if you're in an environment where keeping your computer open to the elements is acceptable I wouldn't try this with young children.

But, as always, I do like.

During college, friend-of-the-blog Patrick had a working computer stuck to his bulletin board. The power supply sat on a nearby shelf, I think, but otherwise, it all hung off the wall.

I'd be curious about RF interference in the surounding area, but this looks like a neat thing.

Trying to install boom

2011-10-06T09:47:00.002-04:00

boom is a key-value store, running on the command line and written in ruby. I don't know whether I need a key-value store for my command line, but I remember when I didn't know if I needed a laptop, bash, Linux, VirtualBox, an RSS aggregator, and so many other things. So, I thought I'd try it. And, I of course decided to try it first on the machine I command line the most, my work machine, which runs Ubuntu 10.04, the long-term support version.

jacoby@oz:~$ uname -a
Linux oz 2.6.32-34-generic #77-Ubuntu SMP Tue Sep 13 19:40:53 UTC 2011 i686 GNU/Linux
jacoby@oz:~$ ruby -v
ruby 1.9.1p378 (2010-01-10 revision 26273) [i486-linux]
jacoby@oz:~$ gem -v
1.3.5
jacoby@oz:~$ gem install boom
WARNING:  Installing to ~/.gem since /var/lib/gems/1.9.1 and
	  /var/lib/gems/1.9.1/bin aren't both writable.
WARNING:  You don't have /home/jacoby/.gem/ruby/1.9.1/bin in your PATH,
	  gem executables will not run.
ERROR:  Error installing boom:
	multi_json requires RubyGems version >= 1.3.6

jacoby@oz:~$ gem update --system 1.3.7
ERROR:  While executing gem ... (RuntimeError)
    gem update --system is disabled on Debian. RubyGems can be updated using the official Debian repositories by aptitude or apt-get.
jacoby@oz:~$

I followed this up by downloading and building ruby and trying to do a gem install from that. Into my second day, I decided that spending more of my precious time trying to get it working on Ubuntu was a waste. So I briefly considered reimplementing it in perl, then found ruby for Windows and installed it on my Win7 box. Then I opened a term and did the gem install. And now it works.

I find this profoundly disappointing.

I don't think this is the fault of Zach Holman, the creator of boom. I have now tried boom on Windows and kinda like it. I suspect (and might pop open VirtualBox and a bleeding-edge distro to check) that the problem is that Ubuntu or just U10.04 is stuck with an ancient RubyGems the way that RHEL is stuck with ancient Perl. I just know that I don't believe it should've been a problem.

2011-10-04T13:17:00.003-04:00

http://www.theatlantic.com/technology/archive/2011/10/for-some-strange-reason-i-want-to-talk-to-my-ipad/246130/

I have headphones at work, which I use for the express purpose of filtering out the sounds of my coworkers, and the sound of them talking to each other is distracting enough. Voice control makes all the sense in a car, but on a tablet?

Tablesorter Bug

2011-10-04T09:41:00.002-04:00

We use the jQuery module tablesorter to allow for the dynamic sorting of our tables, and I have just now run into a bug.

We have 3 second-generation sequencers: the 454, the SOLiD, and the HiScan. We have a second-generation sequence table, which mingles all three. And sorting on sequence agent is broken, because it trys to look at the column numerically when clearly, it needs to treat them all as character strings. Which will monopolize my next few hours.

Learned some Javascript (and maybe more) (I think)

2011-09-29T15:52:00.001-04:00

var filename = get_filename_or_something() ; # !important
if ( !! filename ) {
    # do code
    }

It took me a while, but I think I get this. If you try this and filename is undefined, that could be a problem. One bang forces it into the logical realm, and the other bang reverses it, making positives negatives. In perl, we'd just go if ( $filename ) {} , but we're cool like that.

But, as you might imagine, it is absolutely impossible to look this one up on Google.

How a technical presentation Q&A session turned my head around.

2011-09-29T13:10:00.001-04:00

Sometimes you learn something beyond what you expect.

Last night, I attended a presentation at my local LUG of Puppet by Garrett Honeycutt of Puppetlabs. Garrett had evidently been a PLUG guy when I was, but I didn't really recognize him. I did recognize his friend, so it could be my mind going.

Anyway, Puppet, in a nutshell, is a configuration manager useful for keeping hundreds of machines doing what you want, using the same kinds of tools people use to manage software, including moving from Development to Test to Production. Some people I know who ride the big iron are finding their current means to manage their hundreds of machines to be a little too hodge-podge and daunting were there, and it seems they like the taste of that Kool-Aid. I was interested, and thinking it could be useful, but a bit more than anything I need.

That's when it hit me. In our current workplace, we're not really doing that. There's one canonical instance of a thing, be it a Perl program, a Perl library, a Javascript module, a web page, a script, whatever. We have test and production databases, and for some of the modules, we're using tests like in Perl Testing, which I must confess I never really "got". So, while we might be generating decent code, we're not engineering it and maintaining it the way we should.

So, I'm spending much of today trying to wrap my head around that, hoping to establish Dev-Test-Production setup for my code. Thank you, Garrett and Mike and the rest, for kicking me the right direction without even meaning to.

It Exists!

2011-09-26T11:23:00.000-04:00

Turns out that "Bluetooth for Location" has been solved already, with BlueProximity. I'm still looking for a way that a recurring crontab thing can be set up, but a precedent has been set and licensed GPL, so I'm optimistic.

Also, apt-getting bluetooth and bluez-utils made the broken little dongle shine blue. Just that easy.

How Can My Computer Know I'm At My Desk?

2011-09-25T23:43:00.000-04:00

The problem is exactly as stated. I have my computer do certain alerting things, but unless I'm physically near, I don't it to do some of 'em. And it would be nice to be able to use it to enable certain security aspects, less "unlock when I show up" but rather "lock when I leave". But first, of course, we must have that mechanism.

So what are our choices?

Motion Sensor - I like it. I used to have one in my den to control the light. It's neat. For work areas, I strongly suggest them for your lights, and have on this blog in the past. But you don't always have motion, especially when you've messed up line-of-sight or have a moment of stillness. So, it's a factor, but not the factor.
Light Sensor - That won't tell me everything, but will tell us something, and that something is that the light switch is on at my desk. Not perfect, because often my coworkers turn the light on when I'm not yet there, but it's certainly better than nothing.
Location Sensing - My phone is the very definition of an Every Day Carry, and being a GPS-enabled Android which talks to Latitude, I can query Google to find out where I am. Except, where I am at work is in a subbasement, with no radio and no cellphone, so the location sensing defaults to a switch on the other side of campus, which hardly works. Maybe I should try it for "at home".
WiFi - If my phone is connected to my home network, my switch knows it and with a little coding, I can use it. Which is great, but when I'm at home, I often have WiFi off, and sometimes I use 4G (in the one corner of town where Sprint has 4G). Sometimes I don't though, which makes it a poor indicator at home. But it gets worse at work. My desktop is behind a firewall and my phone would be on the campus WiFi, so never the twain shall meet. I could write something that announces every 10 min or so that I'm on the campus network, should I ever get far enough into coding Android, but it isn't there yet.
Bluetooth - This is the one I'm currently dreaming about. My dongle is Class 2, which means I have about 30 feet of range (and possibly less at the office). That's a nice range.

Without the addition of hardware I don't yet own, I'm thinking that the best solution is Bluetooth. I think that I could have fun making a dingus with Arduino that does motion sensing and light sensing (and maybe a thermometer) so I can get all sorts of data I can play with, but I can't play with that at work tomorrow.

Any thoughts I'm missing besides USB dongle and smart card you can think of?

The Delicious Fruit Has Become Bitter

2011-09-15T22:51:00.002-04:00

I just got mail from Yahoo.

It's the opt-out email. YouTube creators Chad Hurley and Steve Chen have bought it from Yahoo! and are going to try to get it going again.

I wish them luck.

I doubt will go anywhere.

I got into Delicious when I was at the clinic, I think. There were two main selling points for the service:

Your bookmarks were saved in one central location, so that you could share them between your work machine, your home machine, your laptop, etc.
You could see what other people bookmarked and found valuable on various subjects.

Point #1 was crucial to me. You start getting computers, you start getting headaches about knowing where your links are. Point #2? I tried it a few times. It never was too valuable to me. When I set up a new computer, one of the first things I did was sign in with Delicious and download the Firefox plugin.

When Chrome came along with their bookmark system, I jumped. Right now, a few games that don't work right in Chrome are all that keep me going back to Firefox, plus the occasional debugging, and I never ever log into Delicious. A few months ago, I downloaded the bookmarks from Delicious and put them into Evernote. I don't think I've surfed from it yet. I doubt half the content is still there.

Pop Culture Rant, not so much a Tech Rant

2011-09-13T12:05:00.000-04:00

George Lucas doesn't give a sh*t about you. It's survival of the fittest, friend, and George has the f*cking editing bay.

I think we saw what we needed to know about George Lucas in Return of the Jedi. We know that Boba Fett's backpack has jets, but he couldn't jet out of the Sarlacc pit. This song is as cool as Fett seemed in Empire Strikes Back, as cool as he's presented here by MC Chris — he'll chase you "from Endor to Hoth, from Ripley to Spock" — but he dropped into the Sarlacc pit like a sucker and couldn't get out even though he had the tools to. Forget, if you can, that Star Wars begins with Jar Jar and ends with Ewoks and just focus on how the intense human drama at the end of Empire gave way to the fairly ham-handed beginning of Return.

Watch the movies. Enjoy them. But remember that Lucas has lost the script. Don't invest yourself in Lucas, because he doesn't care about you.

Password Strangeness

2011-09-09T09:41:00.002-04:00

I'm in this group. This group has a web tool. This web tool is made with ease of use and trust as cornerstones, so there's one password. That password, for purposes of discussion, is Abc123. Be it noted that I remember the letters and numbers, but not necessarily the cases.

While waiting for the bus to work, another member of the group wanted access to the web tool, and emailed me for the password. I did some checking on my Android phone and told him that the password was ABC123, because that worked with my phone.

When I got to a computer, we did some testing and found that neither abc123 and ABC123 before I checked the password configuration and found the right one.

So, why, when I put the password into this tool, did it take ABC123 as Abc123.

Configuration Joys

2011-09-08T10:51:00.001-04:00

I just got a new remote. I don't suppose I really needed one. I have three in my bedroom (four if you count the wireless keyboard) and I'm more than fine with handling them. It wasn't "Hey, I need a new zapper in the bedroom" that lead me to jump on this.

It was my RSS feed.

I have Radio Shack stuck in there, mostly to see when Radio Shack starts selling the Arduino. What I saw was an announcement of Logitech's Harmony line of remotes. I had heard about them. A friend has a fairly advanced one that has a screen and allows you to combine button-presses for several devices into one action, such as "play movie" starting the DVD player and moving to the composite input.

I got the Harmony 200, a much less complex and cool remote. Also, at $20, much much cheaper. It can control three devices, which works well for the bedroom (TV, DVD, Cable) but the living room has TiVo too and the four-device Harmony 300 is $10 more.

The selling point, the thing that made me go out and get it, was the programming. Well, the price, too, but that just meant it was possible. You plug it into USB and there's a quick-and-easy screen which lets you tell it this is my TV, this is my DVD player, etc. You can then customize the buttons. Then you sync the settings and you're done. No more messing with the "Press the Mode button for 4 seconds, look for the flashing LCD, then punch this four-digit number in" or the worse "Press Power then Down again and again until the TV goes off".

I'll mess with finding different combinations, but this is a solid, well-made remote whose ROM I flash to bend it to my will.

We live in the future.

More Fun With SQL

2011-08-16T15:02:00.003-04:00

I've mentioned this bit of code recently:

        INSERT INTO accession_analysis (
            accession_id ,  analysis_id ,
            reference_id ,  status ,
            status_text  ,  extra_parameters
            )
        SELECT
            accession_id ,
            ? , ? , ? , ? , ?, ?
        FROM accessions
        WHERE request_id = ?
        ORDER BY accession_id

Well, it's slightly different. Before we were just holding the current state ( waiting, working, success, failure ) and now we're holding text information, too, and you don't just want the current state, you want to be able to look back. So, in addition to accession_analysis , we're adding accession_analysis_status, which will have, so far, a unique id, an id for the AA it connects to, and then the the status information.

And the run now. I have to add a run number to the accession_analysis schema. I can do that, but that's not germane right now.

What is germane is how to store the status into accession_analysis_status at about the same time as it goes into accession_analysis, getting the id from the accession_analysis table. It would be far easier if I was using an iterative approach, but then I'm blasting the DB with many connections instead of just one.

An approach would be to find all the accession_analysis elements without a matching accession_analysis_status, and then inserting them into accession_analysis_status. Something like

INSERT INTO accession_analysis_status ( 
    aa_id , status , status_text 
    ) 
SELECT id , status ,status_text 
FROM accession_analysis 
WHERE there's no aa_id corresponding to the id in accession_analysis

But clearly, I don't know how to express this as SQL yet.

Command Line Comments

2011-08-15T03:38:00.001-04:00

Just read Life on the Command Line, wherein the author explains how he does everything everything everything on the command line these days

I'm largely sympathetic. Honestly. And I'm close. Look at my work Linux box and you'll see a good half-dozen terminal windows open. I'm a programmer whose two primary programming styles are command-line/batch/crontab and web programming, so I am much more comfortable with those styles and their explicit order than the GUI developers.

But I have sort of a guilty secret.

I spend most of my programming time using KomodoEdit.

I have a fairly custom .vimrc that allows me to do cool things in vi, but beyond crontab editing and other things that look for $EDITOR, I only use vi when I want to do lots of specific and repetitive find-replace stuff or need to do small changes, or when I'm using SSH to connect to a system and not using SSHFS to mount it. My heavy lifting for editors is done via KomodoEdit.

And that isn't it. Not by a long shot.

I have a significant amount of music. I used to use Rhythmbox as my media player because it allowed me to use and thus alias a command-line interface. On that system, I would have rhythmbox-client --play ,rhythmbox-client --pause , rhythmbox-client --previous and rhythmbox-client --next aliased to play , pause , prev and next .

I stopped for a few reasons. I now have a Windows machine that I use for testing web dev in different browsers, and, when I play my media, I play via Windows Media Player, in part because playing and queueing media that I have but didn't have in my media library made Rhythmbox unhappy. But, between Amazom, Google, Rdio, Pandora and Spotify (which I love), I hardly listen to that huge library anymore, because either I have much of that library up already or I'm streaming stuff I want but don't have. And those tools don't give me a command-line option.

It gets to what tool is powerful enough to allow me to do what I need to. With music, I need it on and going, and paused on occasion.

Don't get me wrong. I'm with him at least in part about mail sorting. Before web tools and Thunderbird, my preferred mail client was PINE, but there's functional reasons that I can't remember anymore that I moved the one remaining command-line mail account from the nmap-based sorting from mh/nmh to procmail. IIRC, I liked the syntax for mh better but I found that SpamAssassin worked better with procmail, and I desperately needed SpamAssassin. The sorting choices I get from both Gmail and Zimbra are both clunkier than a nice mh or procmail script. But I spend little of my time in email, so that shouldn't make much of a difference.

Anyway, interesting read.

Fun with SQL

2011-08-10T16:54:00.000-04:00

If you want to put one thing into a database, that's easy.

 INSERT INTO table_1 (field_1 , field_2 , field_3 ) VALUES ( 4 , 1 , 2 ) ;

I've been putting lots of things into table_1 that are in table_2, where there are n instances of table_1 in each table_2. This lead me to something like this.

 
my $sql ;
$sql = << 'SQL'
SELECT id FROM table_2 WHERE field_3 = ?
SQL
my $ptr = db_arrayref( $sql , $param->{ field_3 } ) ;

$sql = << 'SQL'
INSERT INTO table_1 ( 
    field_1 , field_2 , field_3 
    ) 
VALUES (
    ? , ? , ?
    )
SQL

map {
   my @vals ;
   push @vals , $param->{ a } ;
   push @vals , $param->{ b } ;
   push @vals , $_ ;
    }  @$ptr ;

That's OK for what it is, but what that ends up meaning is lots of small SQL commands sent to the server. SQL handles the commands well, but the opening and closing of network connections is just sort of sucky, so, that's not optimal. So, if you can just send one command, that's what you want. And in this case, the command is this:

 
INSERT INTO table_1 ( field_1 , field_2 , field_3 ) 
    SELECT 
    1 ,
    2 ,
    field_3 FROM table_2 WHERE field_3 = ?
    ;

Isn't SQL wonderful?

Failure to Plan is Planning to Fail

2011-08-01T15:02:00.000-04:00

I have a database table full of requests. When that table is filled, we also put together a wiki page for that request, because the the data we generate for each request can be fairly free-form. This means that we effectively divorce the state of the DB for a given request and the state of the wiki.

There are certain fields that are on the wiki page and not the database. Specifically, contact information for the person making the request. It made sense to me, at the time. I think it was based upon the data separation issue, or that we actually hold that in a profile, too, but I can't remember right now.

Now, my boss wants to be able to regenerate the wiki page for quick-and-easy refresh. Regenerate the DB stuff into a new wiki page and cut-and-paste the non-DB stuff from the old page to the new page. Easy-peasy. Except, I don't store all the data in the database.

So, now, I'm looking at adding columns to the table to handle said contact information and spidering the contact info off the wiki pages. If I had just put it into the database in the first place...

More Javascript Didacticisms: Namespaces

2011-07-29T16:10:00.002-04:00

For the draft page in the Blogger dashboard, I'm seeing sixteen javascript libraries and five extensions running in Chrome. That's a lot. Think about that next time you type var scratch = "something" .

If you want to use scratch and so does the person who wrote the popup library, your code might not work. I learned this when I was writing test code to properly create objects for jQuery's post(). I was sharing a variable that contained the URL for the JSON I was grabbing, and when I had Before commented out, After worked perfectly, but otherwise, it all crashed.

The solution? Objects.

var my_ns = {
    foo : x' ,
    bar : 14 ,
    blee : function () {
        ...
        }
    }

$(function() {
    my_ns.blee()
    } )

Thank you, Stack Overflow!

2011-07-28T14:57:00.003-04:00

I have to thank Stack Overflow again.

I code everything from SQL to Javascript, and that means that I don't really get to become expert in everything. I do alright, but there's occasionally things I just don't get, and I don't really have a community of developers around me to ask.

My most recent problem relates to hashes in Javascript connecting to JSON. You can make arrays behave like hashes in Javascript:



var hash = new Array()

hash['foo'] = 1

alert( hash.foo )

My problem was with post, the jQuery method to do AJAX with large data sets. This didn't work.




$.post( url , hash , function() { ... } , 'json' )

Because hash is an abused array, not an object. Before I figured it out, today, thanks to SO, I make a long string that looked like { 'foo' : '1' , 'bar' : '2' } and ran eval on it. Bad bad bad bad bad. I now know that it should work like this:



var hash = {}

hash['foo'] = 1


$.post( url , hash , function() { ... } , 'json' )

Thanks again, StackOverflow!

What to do with massive amounts of computrons, plus quitting in front of people

2011-07-27T12:29:00.001-04:00

I am in the bioinformatics space. Irony is, there's no way I could've paid to attend OSCON.

Dialup Slowdown

2011-06-30T10:31:00.000-04:00

This is the dialup handshake, slowed down 700x.

I used to have a 2400 baud (call it ~2Kb/m) and I'm now getting 5MB/s, which is more than a 700x speedup.

My .perltidyrc

2011-06-29T13:05:00.000-04:00

Especially when working with others, it is important to be able to format your code. Perl Best Practices convinced me to start using PerlTidy, and this is my current .perltidyrc.

# my .perltidyrc config file
#   http://perldoc.perl.org/perlstyle.html

  --backup-and-modify-in-place
  #--cuddled-else
  --continuation-indentation=4
  --indent-columns=4
  --maximum-line-length=78
  #--line-up-parentheses
  --opening-brace-always-on-right
  --noopening-sub-brace-on-new-line #  -nsbl        # opening sub braces on right
  --indent-closing-brace  # thought this was -ibc
  --indent-block-comments #  -icb         # indent comment blocs
  --vertical-tightness=0
  --vertical-tightness-closing=0
  --closing-token-indentation=3
  --stack-opening-tokens
  #--stack-closing-tokens
  --space-terminal-semicolon
  --space-for-semicolon

  --brace-tightness=0
  --paren-tightness=0
  --block-brace-tightness=0
  --square-bracket-tightness=0
  --no-outdent-long-quotes

I Keep Telling People!

2011-06-27T23:35:00.000-04:00

This guy on Reddit points out that, in 1996, a simple statement of many of our regular lives would be taken as science fiction.

Mary pulled out her pocket computer and scanned the datastream. It established contact with satellites screaming overhead, triangulated her position, and indicated there was an available car just a few blocks away; she swiped her finger across the glass screen to reserve it. A few minutes later, she spotted the little green hatchback and tapped her bag against the door to unlock it. "Bummer," she said as she glanced at her realtime traffic monitor. "Accident on the Bay Bridge. I'll have to take the San Mateo. Computer, directions to Oakland airport. Fastest route." Meanwhile, she pulled up Kevin's flight on the viewscreen. The plane icon was blipping over the Sierra Nevadas and arrival would be in half an hour. She wrote him a quick message: "Running late. Be there soon. See if you can get a pic of the mountains for our virtual photospace."

This is not a new thought. I worked in a used bookstore during my first sojourn in academia, around the summer of 1991, five years before the point in question. At that time, I was big into cyberpunk science fiction. Bruce Sterling and William Gibson. I was putting things away, cleaning up the place, when I saw a book centered upon a computer hacker. It wasn't in the science fiction section. It wasn't even in the true crime section like Cliff Stoll's The Cuckoo's Egg. It was in mystery. That was the point where science fiction stopped being about futurism for me.

Lovin' Me Some Gingerbread

2011-06-12T00:23:00.000-04:00

So, last night, I forgot to plug in my phone, which is an HTC Evo, an Android phone. I've had it since November, and if I forgot the overnight charging, I'd have a dead phone.

I had something like 77% charge this morning.

I love the Power Management under Gingerbread.

Remixing for Cache

2011-06-10T10:18:00.001-04:00

I have a web thing. I did the server end stuff in Perl and right now adding some UI usefulness stuff with Javascript. I'm doing this using Chrome as my #1 browser, under Windows 7 64-bit, and I'm finding a problem.

Specifically, I make changes in the .js file and can't get Chrome to load the changes. I know that I can get that with CTRL SHIFT DEL , but with Firefox, if you hit CTRL SHIFT R on Firefox (and I think IE, but I've never been a big IE user so I'm not sure), and I somewhat recall that behavior as something Chrome did but no longer does.

Anyway, GRRR.

Anger Subsides

2011-06-09T16:14:00.000-04:00

I don't know if it was Sprint getting it's act together.

I don't know if it was Google getting it's act together.

I don't know if anyone from either saw my blog post and felt my anger.

I don't know if moving from Froyo to Gingerbread made all the difference.

But I have now fully connected my Sprint phone running Android to Google Voice. And I am happy.

Want To See Something Cool? or Why I Wish I Was A Hardware Hacker, Pt 2

2011-06-04T03:46:00.000-04:00

Follow this link.

Glasses-mounted display. Which is connected to a 1GHz Beagle Board, powered by batteries to make it mobile. Man, that's cool.

I thought about linking the image, but I'm not seeing "Creative Commons" anywhere on there.

Following the links to the guy's current blog, I saw that it's relatively simple to solder on an RCA cable, which makes the composite link to the myvu glasses. I'm thinking you could get readable console mode with that, which is good. Wouldn't want to try a windowed interface.

The Future of Desktops

2011-05-23T09:55:00.000-04:00

I don't know that I agree 100% with this assessment, but I can say that I agree at least 80% with it. The new form factors of laptops, netbooks, tablets, game systems and smartphones are doing most of what the desktop was doing. You want to browse and write something at the desk or do you want to do it on the couch in front of the TV? (The TV has it's own set of computing additions, as I've covered previously, so I won't go in depth here. Except to say that a friend has a Samsung Smart TV, and my goodness, what a sexy bitch it is.)

Here's the thing: I get my money to buy post-PC gadgets by doing one of few jobs I don't think you can really do without the PC. I am a programmer, and I have two PCs at my desk and three monitors. And I do use them. I have virtual desktops on each computer so I can switch between what I'm seeing on each. If I'm doing web development, I tend to have a web browser with documentation on one, editors on the other, and the page I'm working on at a third. One 13" or less screen isn't going to be able to take the place of 3 21"+ screens.

I like the modularity of desktop machines. You can open them up and tell that this is the power supply, this is the hard drive, this is the motherboard, this is the CD drive, etc. If you know which part I think I touch on how archaic this thought is by my using "CD drive" and not DVD drive, Blu-Ray Drive or Optical Drive. A laptop isn't. If the screen goes, if the drive goes, if the USB port is busted, your only choice is to get a new one. It is, in engineering terms, a black box. You know the outputs, you know the inputs, but you don't have an idea of what's going on inside or how to change or fix it.

A Black Box

Another Black Box

Not A Black Box

It is clear to me that, for most things, the desktop is a dead thing. And for most things, I don't mourn it. I for one welcome our new gadget overlords. I love having a screen in my car, streaming music and giving me directions. I love having our TV get content from Netflix and That Guy With The Glasses as well as NBC and Comedy Central. I love being able to go from "The guy who played Ted Bundy in that movie" to "Mark Harmon" with a quick search of IMDB, without having to get up from the couch. I like it all and I want more.

But the guys who tell the things how to do what they do will be sitting at a keyboard behind big screens for the foreseeable future.

Unboxing Porn - Lenovo ideapad

2011-05-20T09:44:00.001-04:00

This is my new laptop, a Lenovo ideapad.

I have named it Gort.

That fits the convention for all my computers.

It's pretty. I like it.

I hope to do some programming on it soon.

Problem with the new Music for Android, I think...

2011-05-19T09:02:00.000-04:00

I like Buddy Miller. I like his new album, The Majestic Silver Strings, with Marc Ribot, Bill Frisell and Greg Leisz. But as much as it seems like something I'd do, I haven't listened to "Meds" full-time for the last 24 hours. Honestly, it does sound like something I would do, but no.

Here's the thing. I've been using the new Music from Google on my Android phone. I have Last.FM set to scrobble, and for nearly everything else (Pandora, old Music), it scrobbled the songs fine. I think the issue is that the new Music, which is still in beta, isn't telling whatever Last.FM reads that it paused or stopped, which means that Last.FM thinks that "Meds" is the Song That Never Ends. Oh so fun.

So, how do I file this bug?

Why I Wish I Was A Hardware Hacker

2011-05-13T15:31:00.000-04:00

In our lab, we have a device that we call the Minus 80. We call it that because it is a freezer that is meant to store our samples at -80°C, or -112°F. For our purposes, as I understand them, this is not as much overkill as you might think, and there are people who get very concerned when it drops below, or rather, when it pops above -70°C. Drag out from the wall and point the big fan at the coils concerned. Conversations with HVAC guys are so loud you can hear them on the other side of a closed door concerned. Last summer, environmental issues surrounding this and other instruments (here meaning "not a computer") were a primary concern that affected use every day.

The unit in question has a digital readout of the current temperature, on a glowing LED mounted at about shin-height, which is not optimal if you're hoping to monitor the temperature. Which leads me to something that I am very sure I saw on the Ben Heck Show, or else on Make. An LED takes a current to light but also four signals to show the number. At least I think so; it's been over a decade since I last tried working with a breadboard. So, given the signal feeding the digital readout, we could run that into something like an Arduino, and from there through more normal means into our database, where we could do alerting should the thing get too warm when nobody is looking.

Anyone who knows me knows that I love taking data in one form and turning it into another. Web to RSS. RSS to JSON. CSV to PNG. If I can get the temperature out of the freezer — the temperature data, that is — I am sure I can do with it whatever I need to. It is that physical step that befuddles me. Yes, I have been annoyed for a good long time about the Minus 80's lack of serial or USB interface. Now, I have a sense of how to I could actually get there myself.

Not that the boss would let me take a soldering iron to his Minus 80. Which is a little more frustrating....

Don't Throw It In A River

2011-05-12T13:17:00.000-04:00

The ChromeBook. It's a neat thing.

As a CS guy, I spent a lot of time in the Sun Lab. There, there were row upon row upon row of then high-end Sun boxes, and a thing with them was that when you logged in on one, you had all the settings and configuration and background and files as you had when you logged on at any other machine, and anyone else who came after you had their own configurations. Since I graduated, I haven't seen that nearly as much.

That's what I like about the idea of the ChromeBook. I use it, log off, pass it to my wife, and her stuff is there not mine. It means that each setup isn't its own delicate little flower, which interrupts our sense of ownership, but that's OK.

I wonder if something like this is how we're going to handle car computing in the future.

Angry at Sprint, Angry at Google Voice

2011-05-03T15:44:00.000-04:00

I am a Sprint user. I have carried an Evo since Thanksgiving.

I have been a Google user for years. I have been a Google Voice user since it was GrandCentral. When I give people my phone number, I give them my Google Voice number rather than the phone's number. I jumped to having an Android phone as soon as I could swing it, for the smarts of it and in hopes that it would integrate with Google Voice.

So, when I heard that Sprint was offering Google Voice integration, I was all over that, signing up and waiting for it to say "Set it up". Which happened today.

Currently, I have a certain amount of integration. When I'm called at Voice, it rings on my hip. When I call, their caller ID says my voice number. So, I'm largely integrated. But there is a number I never use, and I'd rather dump it and always use my Google Voice Number.

Unfortunately, Google Voice cannot be enabled on this Sprint phone.
Please contact Sprint for more details as to why this cannot be enabled.

So, I called.

Evidently, the problem is that the HTC Evo is incompatable with Google Voice. They didn't really get into how, but honestly, that seems absolutely stupid. They did suggest I upgrade my phone, but I have a great phone and I don't really see anything I want that I don't have now.

Which makes me wonder what integration even means.

The Down Side to HTTPS Everywhere

2011-04-19T15:03:00.000-04:00

HTTPS Everywhere is a project of the Electronic Frontiers Foundation (EFF) that seeks to encourage the use of encryption on the web, including a Firefox Add-On that changes links to HTTPS for sites such as Twitter and Facebook. I've mentioned this before, and I still think it's a great idea.

Google has released an encrypted site, https://encrypted.google.com/, where your queries and responses are protected by encryption. But they don't support it for Images and Map queries. And I think I know why.

A browser can only have n connections to a website at one time. I think that's generally eight. This means that, if you have an image-heavy site, such as Google Image Search, you can only serve the page, the Javascript, the favicon, and five images, and you have to wait until that's done before you can start working on images six through thirteen, and you need to have all the images there before you can render. The trick is to have a Content Delivery Network (CDN), or a bunch of other servers to serve the images. That way, you can download eight at a time from each node and get 'em as fast as the network will let you.

But, if you have an HTTPS-served page, the browser considers everything not coming from that server as a potential breach, as well it should. This means that CDN is a huge problem. That, I think, is why Google isn't using HTTPS Everywhere.

Streaming Problem

2011-04-17T10:06:00.001-04:00

I run Boxee on two machines. First is a fairly modern Vista-running machine, connected via wired ethernet to my modem/router/switch/access-point. The second is a Dell Optiplex running XP that was being retired from university use when I got it, connected to the network with a Netgear USB WiFi dongle.

I have demonstrated that I can serously hinder the ability of the second to stream, simply by surfing the web via my netbook. I don't know which part of the chain is the weak point, and even the strong points are not strong on a PC that old. My friend Patrick says that running CAT6 through the home is kinda worthless for most people because home broadband bandwidth are and will be much smaller than even older WiFi protocols. I understand this and believe it, but problems like this make me wonder.

Do People Really Want Geolocation?

2011-04-13T22:27:00.001-04:00

Reading 2011: The Year the Check-in Died on ReadWriteWeb. Not too sure what to think of it. Of the people I know, only two people who live in the same city as I am have used any of the check-in services I've dealt with, and for neither was it often enough for the serendipity thing ("Hey! You're at the mall, too? How are you doing?!") to happen. I have yet to find another person who wants to share their lat and long via Google Latitude with me. And, I must admit, there's only a small set of people who I'd do that with.

The author lists four reasons why people check in to places — Finding people near you, Rewards, Remembering things and Personal Branding — then proceeds to explain why this is trivial and not lasting and nothing to base much on. There's just not much in it for people to tell others where they are.

But I want to know where I am. Or, rather, I want my desktop, home of my alerting crontabs, to know that I'm near my work, where it knows to alert me like this, or near my home, where it knows to alert me like that, or somewhere else, where it can alert me with a whiffleball bat. Or something. I don't need my friends to know where I am. I don't really want to tell advertisers or employers where I am. But it would be a great boon to me if I could tell where I am. Screen-based alerts are useless if nobody is at the screen.

Be aware that Xerox PARC, the Palo Alto Research Center for the Xerox Corporation, invented computing as we know it. One of the things they had were badges that could tell where you were in the facility, so if someone called for you, the telephone nearest to you would ring, no matter where. That's great and all, but today, everyone has their phone on their hip, so standard wired phones are more and more useless and less and less used.

When I used to work for the medical clinic, I tested badges that could unlock a computer when you got close, and unlock it when I walked away. It seems like it would be easy to write authentication modules that would allow the same thing with Bluetooth, but I'd rather have GPS rolling than Bluetooth most of the time.

There's a Latitude API, but I haven't done more than poke at it. But I suspect this sort of thing will be the real use of location-based computing.

Proper Uses of Social Media: Conferences

2011-04-13T00:57:00.001-04:00

I heard a story once, about a decade ago. A guy comes in to present on Deep Technical Subject A to a conference room full of people. They all had laptops, and the guy thinks, more and more as his talk goes on, that he's dying, that they're all there by fiat and everyone's just sitting and browsing and hoping for the presentation to be over.

He goes through his slides, giving it his best, and gets to "Are there any questions?"

And the questions are good. Well thought out and to the point. Turns out that everyone was on IRC and discussing the points of the presentation as he presented.

We still have IRC, but you can still wear bow ties and spats if you want. Today's thing is Twitter, and many conference suggest a hash tag by which you can follow and discuss the current goings-on. I've seen it suggested for a few things, and I've been disappointed in the past. I can only hope that it's a flyover-country thing and bigger events in bigger areas have more vibrant tweets. Purdue's BoilerWeb 2011 had some great discussions on the #bweb11 hashtag, and there was some nice interaction at Indiana Linuxfest.

I can understand if people don't want to dive into a stream of people mindlessly photographing their meals for everyone to see, but here is a case where it really makes sense.

Podcasts and Synergy

2011-04-13T00:38:00.000-04:00

Image from vanRijn

I've put Google Listen on my phone, listening to podcasts while I drive. One of them is Spectrum from the IEEE, and the one that came on my player was about the Kinect. I have never used the Kinect, as I'm not much of a gamer. Sometimes I'll sit down and play the Wii with the boys, but I have used it more as a Netflix player. So, I'll never use it as intended.

But the interview is not about intention. It is about capability. The Kinect is able to multiple identify people, both by voice and by vision, in noisy, chaotic environments. People are already starting to hack the Kinect to allow Minority Report interfaces and the like.

The second thing I heard was on Hacker Public Radio, which I started to get into after Indiana LinuxFest. The podcast itself is a pre-interview discussion between KDE spokesman Aaron Seigo and Jonathan Nadeau of Frostbite Systems.

Accessability questions have been floating in my head recently. There's lots of buzz about Section 508 in the circles I float through, enough so I've attended two compliance talks in 2011, and my wife has been trying to be a productive person while having severe pain in her right elbow and forearm that keep her from being able to type. She's been playing with speech recognition software, finally trying the Windows 7 native setup that came with her new laptop.

But honestly, when thinking about accessability, I wasn't really thinking about accessibility. I think that there's research at the end of the line for this train of thought that will serve to help accessibility, but it isn't foremost in my head here.

I was driving. I was thinking about cars.

As I think I've mentioned, I've been looking at some travel applications. I tend to have either Listen or a media player of one sort or another playing when I drive, rather than CDs or even worse, radio. (I have only one objection to Amazon Cloud Player, which is that it wants to run only on WiFi, and by and large, I want to use my phone as a media player when I'm mobile, away from WiFi, but maybe the coming of 4G or the next software fix from Sprint will fix that.) I have a TomTom GPS showing my location and speed and I sometimes use Sprint or Google for directions, too. Also, I've been looking into Vlingo, which gives a voice interface to phones. I noticed a bug, hopefully fixed soon, where I hit the Talk Now button while listening to a podcast, and told it to call my wife. Normally, when you tell it to call, it shuts up the media player, Listen included, but here it didn't. So, I'm trying to pause the player and/or mute the player, while on the Interstate.

I know. I know. The point of the exercise is to keep from having this sort of attention-sucking frustration from occurring while I'm driving, because I don't want to hurt myself, my stuff, or other people and their stuff, pretty much in that order. Haven't been too keen of Vlingo since. It's cool enough, but you shouldn't have to press a button while driving to say "I wanna talk now".

And I don't think the Vlingo is alone. Watch just a few high-end car reviews on Top Gear and you'll know that the UIs for even fancy cars are crap.

Ultimately, to do a driving interface right, you have to assume that there will be no vision at all. Most of the things you do while driving (besides avoiding obstacles and other drivers) occur without vision: you use your sense of touch to feel the vibration and you hear the engine to know when to shift. You see the red Check Engine light but you take it in when it starts to sound or feel funny. You can even get away with not often checking the speedometer by staying at a similar speed to your fellow drivers. If you're going to do much computing while driving, it'll have to be via voice. And once that shows up in high-end vehicles, that browser is going to rival Firefox and Chrome and IE at the top of the browser food chain.

(As a pure aside, I think Opera will go there. I don't like Opera and haven't for over a decade, but I know I'm biased on the subject, but it seems like a place that Opera would go, more than anyone else.)

But as we needed the keyboard and then the mouse to get the WIMP interface for computers into the office and the home, we need to rock a voice interface to get the computer into the car. Which is where the Kinect comes in. I kinda think that the car will be a big docking station for future computers, as seat settings and address books and the like are highly personal but a great deal of hardware is there for anyone who sits down. Kinect-based technology would be necessary to distinguish between the driver, the passenger in the front seat who talks with his hands and the loud kids in the back seat. The recognition between conversations between user and computer and conversations between driver and passengers is crucial, and one I think that Microsoft Research and Kinect are closest.

There's going to be a small LED screen for the backup camera in the New Car's rear-view mirror (and some cars of today already have it) and a Kinect as well. Of this I am sure.

Comments on BoilerWeb 2011 #bweb11

2011-04-08T11:36:00.001-04:00

I took a paid vacation yesterday and attended Purdue's BoilerWeb conference. I had a blast. I saw old coworkers and classmates and bosses and had a good ol' time.

Last year, evidently, it was a one-track thing, but interest was such that there were two tracks. This leads to some minor minor complaints. First was the lack of electricity available. For the first track, this was remedied by an attendee (who shares a job and workspace with one of the organizers) bringing in some power strips from the office, but the orientation of the other room and the modular setup of the floor meant that there was only one jack available at the back of the room. If you're going to follow along, liveblog, tweet and such for an all-day event, you need available power.

But that is my only complaint, and I can only categorize that as a minor gripe. And unlike the Indiana Linuxfest, I felt the printed and online scheduling information for this conference was top notch. (Not a slam, ILF guys. Just a pointer on what can be done better next year.)

First pair of presentations were Know Your Limits, covering load testing, and jQuery Plugins. I felt that jQuery Plugins was much more in line with my needs. I write Javascript. I try to write jQuery, which I learned about from a co-worker, but I'm much more the JS guy now and I mostly have experience with my own code, so I don't know the best practices, and have a great tendency to spend time writing my own stuff instead of using used and tested existing plugins. Plus, as I figured out, I kinda clog up the existing namespace, which isn't good. The presenter was very effective, and funny. I really liked his intro use of "Final Countdown" by Europe and high-fiving much of the audience, cheesy at it was, and the presentation showed him eating his own dogfood, so to speak.

Next choice was Introduction to Responsive Design vs NanoHUB. I've know people working with NanoHUB for a while" and while I respect them and their work, it is really orthogonal to anything I expect to ever work with. Meanwhile, Responsive Design as presented was all about writing once and presenting from everything from the desktop to the smartphone with three simple design aspects: Flexible Grid, @Media Queries, and Flexible Images. The coolest part of that, the part that works best with me as a developer, is @media queries. Traditionally, that's used to differentiate different style sheets based upon whether you're looking at it on the screen or printing it. Now, you can put that into the stylesheet and not have to define it in several pages, and make distinctions based upon windows size IN THE CSS. So, one stylesheet. That's the win.

Next was Introduction to Content Strategy, which in a way is fundamental and in a way is meta to this sort of conference. "Content is king" but we spend more time and more money creating the context than the content. We create a framework and say its done, but it isn't done.

My first degree was in Journalism and Mass Communication, and as part of the degree, we were required to get an internship. There was a requirement that set our program apart at that time: it had to be a paid internship, because if the internship was unpaid (as the majority of journalism internships were), there was the assumption that you would not be used for real work and thus you wouldn't learn anything. Years later, as I was starting Computer Science, there was a course where they brought in people from industry to present what you could do with a CS degree. One of the first presenters mentioned that he had internships and that you should send him resumes. I asked if they were paid internships. Imagine if I had asked if we would be breathing oxygen and using electricity to light the office. That's the kind of response. As a society, we do not value the content creators like we value the interface creators. This talk kinda covered that, and offered some means to run the process to make and keep your content current.

There was a presentation on Food For The Heart, which had Ruby on Rails and data munging aspects I was curious about — need to create a test server and learn some RoR stuff — but instead went to the Class Hacks presentation on Mixable. Best way to describe it is kinda Facebook for Classrooms, and it does connect to Facebook. By the tweeting of people who saw the "Food" presentation, the main take-away is that Fat Secret has a nutrition API. This is something I should take a look at. What has my curiosity is Mixable, which I've started to get into for the community of it. Also, a little bit of inspiration in terms of design. I do web design like a web developer: without a real eye for aesthetics and with a tendency to implement new aspects of CSS only until I understand all the controls and get bored. Anyway, Mixable is cool so far.

The talk on Rapid Development in Groups was like being told to eat my vegetables. I know I should start using revision control. I know I should look into MVCs. I know I should eat my vegetables. Being told is not as useful as one might think. The talk Design Patterns Every Web Developer Should Know sort of fit into that. Design Patterns is a subject I have heard about, but don't really understand well enough to know why I should spend my valuable time diving deeper. I've started to look into the three presented, but the presenter had time and audience to dive further into each of the three, so the opportunity was a bit wasted, to the point where "Understand the Facade patttern" has joined the vegetable list, too.

All in all, I think I'm energized and more curious about the possibilities of the newest generation of web technologies. Great conference, guys. Make it better next year.

Why does it all have to be so terribly loud?

2011-04-06T13:36:00.000-04:00

CSS Table Printing Problem

2011-04-05T13:54:00.000-04:00

So, I have a page that holds lots of tables. These tables are all the schema for a database I use for a project. All 31 of them. The page is basically the DESCRIBE table for each table. We're talking usually about 4-8 rows per table. A main reason for creating this page is so we can have the schema taped to the wall.

The problem gets to be that of widows and orphans. In layout, a widow is the last line of a paragraph occurring after a page break, while an orphan is a first line of a paragraph occurring before a page break Or, in this case, tables. And it is considered to be bad.

A widowed row.

Using THEAD and TBODY allows the browser (in this case, Firefox on Linux) to repeat the head of the table, which is nice, but one row of the table isn't enough.

Clearly, these are short enough that forcing the browser to keep the table together through a page break shouldn't be too wasteful, unlike if this was a SELECT * FROM table on a table with hundreds or thousands of rows. But it seems this isn't possible with CSS. CSS has page-break-before, page-break-after and page-break-inside, but they are insufficient. You can force a page break before or after an element, but that's not it. If you can fit seven or eight tables on a page, why would you not put them on a page? And there's three options for page-break-inside: avoid, auto and inherit. The choice of auto translates to "break it anywhere", while avoid has all the practical use of "break it anywhere" and inherit simply means "do what the parent did", which of course means "break it anywhere".

There are CSS properties for widows and orphans, but they don't seem to work for tables. Well, less with tables than tables wrapped in DIVs.

Notes on Indiana Linuxfest

2011-03-28T16:58:00.000-04:00

Last Friday, Saturday and Sunday were Indiana Linuxfest, and I had the opportunity to attend Saturday. I got there in time to catch most of the opening keynote (9am) so I was a bit too wiped to really do the party and see dual core, which saddened me, but I had to be not-sucky on Sunday.

Here are some notes:

The flyer should have a venue map so you know right off where you're going. The talks should be organized by day, and if there's going to be a presenter bio, the talk should have the presenter's name on it, so you know who is giving what talk.
First presentation I saw was on Open Hardware, and I came in late. This sucks, because it was cool. Talked some about the Arduino, which I really need to get, because they're cool. Also mentioned were Beagle Boards (I think), which is cool because they're computers the size of a 3.5" floppy disc. Also, the EZ430, a programmable watch from Texas Instruments. This is everything I thought the programmable Timex Iron Man watch was going to be. His last line was "We live in the future; We might as well start taking advantage of it." A man after my own heart.
Dan Klein, a Googler from Pittsburgh, gave a talk called Frank Lloyd Wright was Right. In this, he told a story about his hometown, where Wright was asked to suggest a plan to fix some of the architecture problems with Pittsburgh. His advice? "Raze it to the ground and start over." Don't just patch and make due with incremental change, but start over and do it right this time. The presenter suggested this should be the way to take on many of the problems with computing infrastructure, as well.
Brian Proffitt talked about "The Death of the Linux Desktop (and I feel fine)". Great presentation by a great presenter, with good discussion.
The Bloomington Hackerspace, BloomingLabs, set up shop at the 'Fest, and that was cool. Hands-on is always cool. I got to pick my first padlock! Yay!

I could go on, but I won't. But I hope they consider this enough of a win to do it again next year, and and I hope and plan to make Ohio LinuxFest in September.

On The Importance of Unique Strings

2011-03-28T11:21:00.000-04:00

I find generalized mail notices to be counterproductive. I get a lot of mail, from many dozens of mailing lists, from many dozens of commercial ventures I occasionally throw money at or want to throw money at, and more. I don't pay attention to it always. I write things to check things and send me email, and even then, I don't often read it. So why would I want to have every little thing beep and pop up and eat my attention when it arrives? Especially when, as a programmer, a lot of problem solving just goes away when that sort of beep draws me out?

So, a while ago, I wrote jBiff, which announces when I have mail (the traditional roll of biff and xbiff) via Jabber/XMPP (thus the "J"). I wrote it so I could configure search strings, so that I could have it biff when I got mail from:

My boss and coworkers - because I can name no worse feeling than hearing my boss come up behind me and asking "Did you get my email?"
My wife and kids
My parents
A select set of friends
The office voice mail
My bank - which tells me when I have run out of money
The Weather Channel - which tells me when there's severe weather in my area. (I work in a sub-basement so I can't just look out the window.)

And it's pretty much in that order. If you're top of that list, I'll get warned within two minutes of receiving the mail, and the bottom get checked every twenty minutes, all thanks to crontab.

So, I was surprised at getting alerted that a boutique amp manufacturer has given away a 2x12" cabinet. I'm not remotely surprised at receiving this email (and no, I didn't win) but I am surprised that it popped up on Google Talk. And it took a little poking.

My eldest son is named "Niel". Not "Neil" or "Neal", but "Niel", as my grandfather was named "Niels" and I wanted to honor my history. And ensure that nobody's going to spell either his first name correctly on first pass, to go with nobody ever being able to pronounce his last name. I can be cruel.

And the representative from the boutique amp manufacturer is named "Daniel".

Daniel.

Because I used my son's first name, not his whole email name, I got alerted. I have since fixed it, but as a reminder to both myself and others, I'm putting this out there: Don't match a short string when you can match a longer, more unique string, or else you'll get false positives.

Now There IS An App For That

2011-03-24T15:49:00.001-04:00

I had to do something. This was something. So I did this.

This program collects the functions by module in a directory full of modules and checks a code base against it. This tells you which functions you are actually using. Which isn't quite what I wanted, but close.

I can see some additions I could want. Setting the library directory and code directories via Getopt::Long would be the first one. And it doesn't quite tell me what I want to know in all cases. If I use a function in a program that never gets called, it still gives me a result. But this is a place to start.

And because of this, I now know that, within the stack of previously-invented wheels called CPAN, there's a module called Regexp::Common that holds a stack of established regular expressions. I wanted to pull out all comments out of my programs for testing, so that a commented-out function doesn't count.

#!/usr/bin/perl

use 5.010 ;
use strict ;
use warnings ;

use Cwd 'abs_path' ;
use Regexp::Common qw /comment/ ;

use subs qw{
    check_programs
    module_list

    decomment

    drop_pm
    get_module_subs
    pull_package_name
    pull_module_name
    pull_sub_name
    } ;

my $modules     = module_list '/path/to/my/lib' ;
my $directories =  [
    '/code/directory/one',
    '/code/directory/two',
    '/code/directory/three', ] ;

my $data = check_programs( $directories, $modules ) ;

for my $mod ( sort keys %$data ) {
    my $module = $data->{ $mod } ;
    say $mod ;
    for my $sub ( sort keys %$module ) {
        my $subroutine = $module->{ $sub } ;
        say join "\t", '',
            ( $subroutine->{ count } ? $subroutine->{ count } : 0 ) ,
            $sub,
            ;
        }
    }

exit ;

########## ######### ######### ######### ######### #########
########## #########     Subroutines     ######### #########
########## ######### ######### ######### ######### #########

#--------- --------- --------- --------- --------- --------- ---------
# The core of the program
sub check_programs {
        my ( $directories, $modules ) = @_ ;
        my $data ;
        for my $program_dir ( @$directories ) {
            my $program_directory = abs_path $program_dir ;
            chdir $program_directory ;

            #say $program_directory ;
            my @directory = glob '*.cgi *.pl *.pm' ;

            my $programs ;
            @$programs = map {
                { $_ => get_program( $_ ) }
                } @directory ;
            for my $program ( @$programs ) {
                my $k ;
                ( $k ) = keys %$program ;
                my $v = $program->{ $k } ;

                #say join "\t", '', $k ;
                for my $module ( @$modules ) {
                    my $mk ;
                    ( $mk ) = keys %$module ;
                    my $mv = $module->{ $mk } ;

                    #say join "\t", '', '', $mk ;
                    for my $sub ( @$mv ) {
                        my $result = $v =~ /$sub/ ? 1 : 0 ;

                        #$result
                        #    and say join "\t", '', '', '', $result, $sub ;
                        $data->{ $mk }->{ $sub }->{ exists } = 1 ;
                        if ( $result ) {
                            $data->{ $mk }->{ $sub }->{ count }++ ;
                            push @{ $data->{ $mk }->{ $sub }->{ used } },
                                abs_path $k ;
                                }
                        }
                    }
                }
            }
        return $data ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# returns the contents of a filename, if it contains 'perl' in the top
sub get_program {
        my ( $filename ) = @_ ;
        if ( -f $filename ) {
            if ( open my $fh, '<', $filename ) {
                my @lines =
                    map { decomment $_ } <$fh> ;

                #return 0 if $lines[0] !~ m/perl/ ;
                return join '', @lines ;
                close $fh ;
                }
            }
        return 0 ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# removes Perl comments
sub decomment {
        my ( $code ) = @_ ;

        #chomp $code ;
        $code =~ s/$RE{comment}{Perl}// ;
        return $code ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# returns an array ref of module names with an array of subroutines
# the module contains
sub module_list {
        my ( $dir ) = @_ ;
        my $directory = abs_path $dir ;
        chdir $directory ;

        my $output ;
        my @directory = glob '*.pm' ;

        @$output = map {
            {
                ( pull_module_name join '/', $directory, $_ . '.pm' ) =>
                    ( get_module_subs join '/', $directory, $_ . '.pm' )
                    }
            }
            map { drop_pm $_ } @directory ;

        return $output ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# returns an array ref to all the functions (minus internal functions
# whose name starts with _) within a given module
sub get_module_subs {
        my ( $mod_path ) = @_ ;
        my @output ;
        if ( -f $mod_path ) {
            if ( open my $fh, '<', $mod_path ) {
                my @lines = <$fh> ;
                push @output, grep { !/^_/ }
                    map  { pull_sub_name $_ }
                    grep { /^\s*sub / } @lines ;
                close $fh ;
                }
            }
        @output = sort @output ;
        return \@output ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# return the package name of a module
sub pull_module_name {
        my ( $mod_path ) = @_ ;
        my @output ;
        if ( -f $mod_path ) {
            if ( open my $fh, '<', $mod_path ) {
                my @lines = <$fh> ;
                push @output, map { pull_package_name $_ }
                    grep { /^\s*package / } @lines ;
                close $fh ;
                }
            }
        return $output[ 0 ] ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# return the package name of a 'package Package::Name ;' string
sub pull_package_name {
        my ( $in ) = @_ ;
        chomp $in ;
        $in = ( split m{\s*package\s*}, $in )[ 1 ] ;
        $in = ( split m/\s*;\s*/,       $in )[ 0 ] ;
        return $in ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# return only the subroutine name from a 'sub sub_name { ' string
sub pull_sub_name {
        my ( $in ) = @_ ;
        chomp $in ;
        $in = ( split m{\s*sub\s*}, $in )[ 1 ] ;
        $in = ( split m/\s*{\s*/,   $in )[ 0 ] ;
        return $in ;
    }

#--------- --------- --------- --------- --------- --------- ---------
# remove '.pm' from end of module file names
sub drop_pm {
        my ( $in ) = @_ ;
        $in =~ s/\.pm$// ;
        return $in ;
    }

Is There An "App" For That?

2011-03-24T10:41:00.001-04:00

So, I've been developing a workflow for quite some time. I have SQL tables feeding data into Perl-generated web sites and JSON feeds feeding AJAX pages using jQuery, and if I think about it, I could probably feed two or three more technology buzzwords into there, too. CSS, for sure.

A big problem is that, as I proceeded, I had no idea what I was doing for a lot of it. What I know about writing Perl modules, I know due to this workflow. Well, I knew "start with Package and end with 1 ;", but beyond that, not a whole lot. The web has helped this process a lot, and now I'm not too ashamed at what I have.

(Object orientation. Moose. It could use improvement and I would learn from the experience. But that isn't today's topic.)

So, by and large, I have modules handling the SQL queries, then passing hashes or hashrefs, depending on what I want. But, as I said, I was figuring out what I'm doing as I go along. I'd get the problem, think "I need these tables", write modules to interface with the tables, write programs that use the modules, fail to write tests for the modules because what tests I could imagine are either trivially stupid or highly dependent on the state of the database (I have Perl Testing on my desk and have worked through the first two chapters, so I know it's a failing, but see the note on object orientation.)

In a practical sense, if I'm dealing with a foo, I will have a table foo and maybe foo_attributes connecting with it. Then I'd write the module Foo.pm with create_foo, update_foo, delete_foo, read_foo and get_foo_list for a list of all the foos in the table. I can tell which tables I'm dealing with from the function name, for the most part. When it gets to the interaction of foos, bars and blees, it gets hairy, but that's why I get paid the big bucks, right?

Right now, I'm looking to document the schema and DROP TABLE cruft before adding a few more features, which means what I'd like to do is look through my code base for all the modules I'm responsible for, search for all the functions I'm calling in each, and give me a list that might look something like this:

list_foo.cgi
- VarLogRant::Foo
  - get_foo_list
  - read_foo

Now, I have a naive text-parsing concept of how I could do this, but honestly, I can't imagine that I'm the first person who would want something like this, and I know that the Perl community loves to build tools for building tools, so is there something I should be looking at before I dive into this myself?

The Schwartzian Mindset

2011-03-16T12:23:00.000-04:00

Yeah, but your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should. Dr. Ian Malcolm

I have recently come to a knowledge of the power of map and grep when dealing with hashes and arrays of data.

The key to entering this mindset is the Schwartzian Transform. Here is a canonical example.

@sorted = map  { $_->[0] }
          sort { $a->[1] cmp $b->[1] }
          map  { [$_, foo($_)] }
               @unsorted;

For years, I looked at that and said "Huh?", then wrote for my $i ( @unsorted ) { ... } or something like that. In fact, let's look at a for-loop implementation that would get similar results.

my @sorted ;
for my $i ( @unsorted ) {
          my $j = foo($i) ;
          $hash{$j} = $i ;
          }
for my $k ( sort { $a cmp $b } keys %hash ) {
          push @sorted , $hash{ $k } ;
          }

That's a lot more code, and it would not surprise me at all if my code was far slower than the Wikipedia example code, but I understood for loops from my first year of programming, so I can look at the second example and immediately understand it, and would still understand it if I came back and looked at it in a month.

Of course, you can comment this sort of thing to explain it to your later self.

@sorted = map  { $_->[0] } # taking away the sorting field
          sort { $a->[1] cmp $b->[1] } #sorting by the 2nd field in the anonymous arrays
          map  { [$_, foo($_)] } # an array of 2-element anonymous arrays
               @unsorted; # the original unsorted array

I find myself using the map and grep method a lot lately. It avoids a lot of extra arrays floating around, but as I looked at the Schwartzian Transform and boggled for years, I could see myself, or the next person holding this seat, looking at the code for hours and failing to get it. But at least, now that I can code like that, I can wonder if I should...

Computing and Simplicity

2011-03-11T11:18:00.001-05:00

As I was preparing lunch the other day, I noticed two co-workers poking at a computer. One looks at me and says "Aren't computers supposed to make things simpler?"

No.

A thousand times, no.

Computers are supposed to allow us greater and more numerous avenues of possibility.

I say that all science is computer science these days because if you can do something without having to handle large amounts of complicated data, using computers to tell machines to manipulate things far smaller than you can do by hand, without using modern instruments and the computers that control them — if you can find out what you want to find out without that, it was probably done years ago.

I say that because computer networking allows for computer networking, for knowing people via Facebook and Twitter and Skype and wikis and chatrooms and whatever, which gives you connections based on interests, not proximity.

If you want simplicity, go out and find a plot of land. Plant seeds. Let it rain. The seeds grow. You harvest and eat the seeds, and plant some. Let that be the whole of your life. That's simple. And it's not for me. The whole of civilization has been built so that more and more people can justify not doing that.

(Lest anyone accuse me of slamming farmers, I have farmers in the family, and they more fully embrace and become more expert in more technologies than anyone else I know. They aren't keeping it simple, either.)

Of course, I said this.

Which is not what people struggling with computers want to hear. They want to hear how to take their means of using PowerPoint to generate HTML and/or images which they then put on the Wiki and make it work, and that philosophic postulates which do not address the problem at hand make them angry.

Of course, I noted that they had fallen victim to one of the classic blunders, the most famous of which is "Never get involved in a land war in Asia", but only slightly less well-known is "Never use Microsoft Office if you can help it."

This time, I wisely kept this thought to myself.

Capital Idea 3 - Going 'Round In Circles

2011-02-25T00:35:00.000-05:00

I thought I had an idea to solve the state capital issue.

Go through each line in the list in order of length
If both points of the line are not handled twice ( connected on either side ), place it into the path
Quit when every point (state capital) is hit twice

This gives you all the points, but it does not give you an order.

pull line out of path
for each line in the path

check if one of the points
if one of the points is the one we want, throw it into the output array
else put it into the dummy path array

push line into path

There is a problem, however.

Nothing I have done guarantees that all the states are connected. And in this case, I know that the states are not all connected.

Back to the drawing board.

Capital Idea 2 - So Far Away

2011-02-18T13:50:00.000-05:00

Rather than query Google Maps to get driving distances between capitals, I found latitude and longitude and use these functions to determine the distance, then ran it against all thousand-some possible distances, so it is ultimately a hash check rather than a calculation to find a distance.

sub haversine {
    my ( $lat1, $lon1, $lat2, $lon2 ) = @_ ;

    my $theta = $lon1 - $lon2 ;
    my $dist =
        sin( deg2rad( $lat1 ) ) *
        sin( deg2rad( $lat2 ) ) +
        cos( deg2rad( $lat1 ) ) *
        cos( deg2rad( $lat2 ) ) *
        cos( deg2rad( $theta ) ) ;

    $dist = acos( $dist ) ;
    $dist = rad2deg( $dist ) ;
    $dist = $dist * 60 * 1.1515 ;
    return $dist ;
    }

sub acos {
    my ( $rad ) = @_ ;
    my $ret = atan2( sqrt( 1 - $rad**2 ), $rad ) ;
    return $ret ;
    }

sub deg2rad {
    my ( $deg ) = @_ ;
    return ( $deg * $pi / 180 ) ;
    }

sub rad2deg {
    my ( $rad ) = @_ ;
    return ( $rad * 180 / $pi ) ;
    }

Programming Challenge: A Capital Idea!

2011-02-18T12:28:00.006-05:00

In the last Purdue Perl Mongers meeting, Mark suggested a challenge. Going from state capital to state capital of the "lower 48" (no Alaska, no Hawaii) , without skipping one and without repeating one, what are the longest and shortest distances. We guessed without knowledge, and then, after the meeting was over, we started working on it.

As you may know, this is a restatement of the Travelling Salesman problem and as such is NP-Hard. There are, in this case, 48! possible answers.

My immediate thought was to specify the most extreme unused distance for each capital, longest or shortest, depending. I also thought to start at Maine, seeing as it's the furthest out of all the state capitals.

You should be able to see some problems. For example, Arizona wasn't the shortest choice until it was the last choice. I'm not remotely happy with the Midwest and New England sections, either. So the fastest choice of algorithm wasn't the best choice. Running the naive longest-path against each state showed that Tennessee and Kentucky were the best starting point, which shocked the heck out of me.

I added some code to pick the closest n and recurse through those, and that cleared up the Arizona problem quickly, but it will take a lot of redoing to get it to back up to the Tennessee-to-Michigan jump, much less the New England jumble. Considering a quick read-through of threading modules, which could help me get through my too-many-possibilities, but also considering ways to rethink this as a breadth-first problem.

I'm also seriously considering just making a web interface and solving it myself. I can't be sure it's the best possible, but I can vgrep for issues and optimizations.

Office Software for an Open-Source-Loving Programmer

2011-02-12T10:21:00.000-05:00

I'm a programmer and IT helpdesk guy for a lab at a major university. I rarely touch office suites. I do, on occasion, prepare presentations for user groups I'm involved in, but as I've found I'm more comfortable diving into whatever applications I'm working with, that's a fairly rare thing. The office tool I most often use is a spreadsheet, which allows me to see CSV data (and we deal with a lot of CSV data) in the formatted way it's meant to be thought of.

For my personal purposes, I've Gone Google.

When your life is spread out among seven computers and a smartphone, connecting your stuff back to the cloud makes more than a little sense. I get the "you don't own your data" arguments, but for now, it fits my needs fairly well.

But it doesn't really fit my office's needs. Genomics data can be big. Big. Loading it up to Google isn't really where I want to go. Which leads to the real crux of this discussion.

I could go Microsoft Office. There was a technical reason that you might prefer Office, in that genomics data can be big, and Excel allowed for 65536 rows by 256 columns, which was more than double what I could get out of OpenOffice.org. However, it seems that this has been changed since I last tried it.

The problem there is that I prefer to go with a Linux desktop, both for the power of it and for the freedom of it, and Office is a Microsoft product that you can't run (without virtualization or something) on Linux machines. Ubuntu comes with OpenOffice.org, so that's my default go-to for those times I do need to play with an office suite product.

But....

I was listening to the FLOSS Weekly show about FOSDEM, and the mention of LibreOffice got me curious. Each time I've opened OpenOffice, I've thought that it was slow and cumbersome beyond what it does. If there's a noticeable improvement in it's speed, it's well worth considering.

You know, for the seven times a year I really really need it.

Named Times in Crontab

2011-02-10T10:44:00.002-05:00

I first dealt with crontab in the late 1990s, and I still run many of those same crontabs. I mean, same program, same times. So, my understanding of that tool is very tied to that time.

Of course, very little changes in technology over 15 years. </sarcasm> So I recently looked through man 5 crontab and found that there are special strings for certain common times.

`@reboot`	Run once, at startup.
`@yearly`	Run once a year, "0 0 1 1 *".
`@annually`	(same as @yearly)
`@monthly`	Run once a month, "0 0 1 * *".
`@weekly`	Run once a week, "0 0 * * 0".
`@daily`	Run once a day, "0 0 * * *".
`@midnight`	(same as @daily)
`@hourly`	Run once an hour, "0 * * * *".

Some of these come with variable utility. What are you doing on a system that's @yearly? Sending "Happy New Year!" to everybody you know? There's much here that strikes me as better done connecting to a calendar. In fact, I'm hatching an idea of setting up a calendar in Google Calendar to do that sort of wide-ranging crontabby or batchy stuff. If something runs less common than weekly, it doesn't get into my crontabs.

Except...

I'm really drawn to the @reboot string. For my work setup, I have FUSE mounts to a great number of machines (which I really use) and a specific line to configure the line-in to go directly to line-out so I can route my Windows box and Android phone through my Linux box to my headphones. I've put those commands into my crontab at @reboot.

Items from the Feed - Tying Down The Uplink

2011-02-08T22:31:00.002-05:00

Video Game - E3 2011 - The Feed

A couple of issues here. First, Sara Underwood is pretty, but that voice! Almost like fingernails on a chalkboard.

Anyway, one of the issues she brings up is wide-open home networks. The warning wasn't folks coming into your WiFi, infecting your PCs and stealing private stuff. It's glomming onto your network and "stealing" your bits.

Honestly, before I moved, I never encrypted my WiFi. I never thought it was worth it. Different neighborhood, with more people closer, and the new access point came with it up by default. As long as they don't use enough to mess up your video streaming, why not?

Pondering the Future, the Shock of the New, and Defining The Terms

2011-02-08T11:23:00.004-05:00

I think I have to start with The New X here. I've gone on about The New Telephone and The New Television without really defining my conception, my mindset. I think it's obvious what I mean with those, but when I hit the next topic on my mind, you get a certain Bob Barker connotation.

Image by Wayne Silver
THE NEW CAR!

I have a concept in my mind of life between 1955 and 1985 (only about half of which I lived through) where the Old Telephone and Television live. The Old Television, for example, consists of complete boxes, with the beginnings of plug-ins from VCRs and Atari game machines. The New Television, still being designed and tested, will involve multiple sources, time-shifting, and modularization such that the screen is but a monitor, one of many.

But TV is fundamentally information. As long as you can see it and hear it, who cares whether it comes from the air, a coax cable or a Cat5e jack? The New Car is different. It is first and foremost a mechanical thing, and thus the issues involved are different.

A person from 1970 stepping into today wouldn't understand cell phones and would be befuddled by the collection of remotes on the TV, but I am very certain she could get into a 2010 car, start it, and drive it around. "Three on the Tree" isn't nearly as popular as it used to be, but the gas pedal, brake pedal, speedometer, turn signals, AC, rear-view mirror would all be about where she'd expect them to be and would behave about how she'd expect them to behave. There would be a few hiccups possible — electric motors and hybrids, OBDII and continuously variable transmissions are three things that come to mind — but they don't particularly affect the driving experience too much. The mechanics of automobiles have been abstracted away from the act of driving, and that's a good thing.

What is changing is the information-related aspects of driving, as fast and as good as the car companies can do it. As the driver of a car that was the cheapest new car we could find, much of what I know is from driving rentals when my car is in the shop, and from watching Top Gear. I see in-dash navigation, but you don't really want to be looking down at the dash too much when driving, and an in-dash system won't have the quick update times of navigating with your phone. Car audio systems have connecting to your iPod and iPhone down just as Android is beginning to dominate the market.

For the car market, electronics is a trailing concern. You want it to be so, clearly, because making the car faster, safer and more economical are the leading concerns. But it makes the choices dated. I saw a video from CNet's Car Tech section where the reviewer talks about the Toyota Camry, mentioning the 6-disc CD player (in the last days of optical media?) and optional dual DVD players in the back seat for too much money, which he suggests you skip and get iPads instead.

I told a friend a few months ago that I wanted to put a computer in a car. He asked "What would you do with that?" The thing that I mentioned is combining OBDII data with GPS so I can connect what I was doing and where I was doing it to what the engine was doing. I'm not a mechanic, so I don't really know what I'd do with that information. That information becomes really usable when you breach the firewall and start connecting to engine, transmission and brake electronics. Currently, you can get aftermarket chips to change the behavior of your engine and such, but that's warranty-voiding stuff, and also, tight inner-loop microprocessor kind of work. It won't wait until the OS pages, queries a database and returns a proper response.

Plus, of course, the standard interface vectors for electronics elsewhere are attention-sucking accident-causing crap for autos.

So, I forsee the creation of The New Car, joining long-term reliable mechanics with high-turnover electronics, as a long-term process that will cause grumbling for a long long time.

Time Estimation Hell

2011-02-07T11:44:00.004-05:00

There's two main tasks on my desk right now. The first is adding a new instrument to one of our web workflows. It's my code, so I'm the one to do the diffs. The second is the Sikuli thing. To recap: We install VirtualBox, create a Ubuntu 10.10 64-bit virtual machine, upgrade libCV to 2.1, install Sikuli, ssh into a Linux compute server to run the Java GUI data modification server, then write Jython in order to script the thing. This will allow us to munge and generate PDFs and such for several things in order. Every time I deal with this program, I feel the nearly-but-not-quite-overwhelming urge to scrap that, decompile the Java bytecode and rewrite the thing in a way that's batchable. I haven't been overwhelmed, so I stick with the Sikuli solution.

I still think Sikuli is neat. I think I have to be clear. It's just that this is becoming a much more complex thing than I am comfortable with.

This is why, when the boss appeared over my right shoulder, I was working on the other one. It's web work. I like web work. It's mostly Perl. I like Perl. I wrote everything involved, from the SQL tables to the Javascript interface bits. I get all of the things I wrote. It is a comfortable place, which means it's a job I can get done and feel productive. I felt I could git 'r done, check the thing off, and then deal with the confusing time sink.

Nothing quite like the boss showing up behind you to tell you that, no, your priorities must change. There's a whole bunch of .DAT files to be munged, and what we have isn't cutting it. So, when will it be done?

No clue.

No clue on several sorts of issues. I started this as a programmer who never did much more than recursive and iterative Fibonacci in Python, so I am not familiar and comfortable enough with the language to really say I know how to do anything fast. I'm still on the learning curve there.

Plus, I don't really know what all the problems are. I don't know how to solve the problems I don't know that exist. We're combining two machines and two Java applications with ssh, X11 and other things. It is a complex thing.

But those things are relatively minor. The more fundamental thing is that I do not think of tasks in time terms. How long a task takes is how long it takes. I don't want to think about the time aspect. It distracts from thinking about how to actually do the task. If I come up with the clever solution, it's done. If I don't, it takes forever.

Is this a common thing for programmers? Or am I the odd duck?

We Live In The Future Redux

2011-02-05T19:53:00.000-05:00

I have my trusty netbook on my lap. In one of my USB ports, I have a thumbdrive with the Ubuntu Netbook Live CD on it. I'm installing Ubuntu Netbook 10.10 right now. But not on my netbook's SSD. I'm installing it on an 8GB SD card.

I remember getting hard drives smaller than that. Storage-wise, I mean. Physically, they were the size of today's DVD drives. I'm sure that the performance of an SD card will bed poor compared to an SSD card, but a second face for $20 is kinda cool. Dual-boot on the cheap. My friend Patrick put Ubuntu on a thumbdrive so he could have "GoodOS" on his work laptop. I wanted to have something that wouldn't hang out too much. I don't use my SD slot often, so keeping it there works for me.

It's still configuring, so I can't say much. But such powerful and modular hardware is neat.

New-To-Us TiVo, New-To-Us GPS

2011-02-05T13:41:00.007-05:00

We have a TiVo. Finally. Yeah, I know we're ... a decade past the curve here? Something like that. I have to add that the TiVo+satellite box sucks because that means only one tuner, which means you cannot record this and watch that. But I have to say, the folks who say that the user interface to TiVo is a cut above? They have something there.

What sucks outright is the IR blasters. Well, they're OK, except there's no feedback loop. It would be far better to have the TiVo be the satellite box. I understand the HDMI interconnections of the Google TV box (of which there's really only the Logitech Revue, last I checked) have bi-directional communication, and that's what you really want.

It's a bit funny, though. We have a PC running Boxee and a Wii that does good Netflix. So far, we're widely distributed. I want a newer, faster PC for Boxee (and other duties, like home file+print server), and with the TiVo we now have access to Amazon On Demand. (Not like we've used it yet. We're still a Red Box family, and spinning discs of optical media at DVD quality is definitely a trailing-edge technology, no matter the shiny red vending machine.)

For Christmas, my father gave me his old TomTom. I have always wanted a stand-alone GPS, but not really for car use. Well, yes and no. I want a list of discrete lat&long coordinates I can map. I'm sure I can have that with the Evo phone, but I haven't really dived into that. Anyway, the point at this juncture is that I have a phone with a navigation feature, my wife had a phone with a navigation feature, and 2 of 3 sons had phones with navigation features. I thought "Hey, I'll never use this".

There are two reasons I've started to. First is the mount. I can mount the TomTom to the window and keep it in a good position. I don't have a good cellphone mount for the Evo. I love having the rolling map and the speedometer even when I don't have a destination set.

The second is duplication. I can have my phone set to do phone things, or provide audio, or even just on my belt as it should, and not worry about it forgetting my destination. Don't know how I'll really integrate it into my driving, though.

New Multimedia Keyboard

2011-02-03T20:24:00.004-05:00

I have the biggest remote I have ever seen.

It's the GKM561R keyboard from IOGear, which I got through NewEgg. It's got a built in trackball, which makes it better for multimedia settings than the Logitech wireless desktop I got before. I have always used Logitech wireless keyboards before, and I'm sure I'll get one again, but the mouse stopped working and got lost, and for multimedia applications, it makes sense.

Image from KVMGalore

There was a weird thing. The W key had fallen off, but it was securely in the packaging and was easy to replace. I'm loving the addition of a TV PC.

The Internet of Things? The Semantic Web?

2011-02-02T14:44:00.000-05:00

I have a dream.

I dream of a lamp. A lamp using LEDs. A lamp that changes color, depending on the ambient temperature outdoors. If the weather outside is frightful, the lamp indoors is blue-to-purple. If Summer's here and the time is right, the light is yellow, orange or even red.

This dream is in part inspired by my workplace in the subbasement of my building. I can very easily forget the season, even, when I'm head-down into my work, so any reminder of the conditions outisde would be good.

This is an example of what was called Ubiquitous Computing or Pervasive Computing, depending on who generated the buzzword. In essence, the data is everywhere around you, not just on your PC.

The thing that's really keeping me from realizing this dream is a lack of electronics skill.

But that's a smaller thing today.

There's a thing called an Arduino, which is an Open Source bit of hardware that begins to open up the world of programming small things. I'm fairly sure I could, given an Arduino and a breadboard, begin to dummy this up.

This kinda gets me to categorization issues. Does this concept, an internet-aware fixture that determines the shade by the weather report fall under the Semantic Web? Or is it more tightly connected to the Internet of Things concept? Or is it a bit of both, like a Reese's cup?

Netflix As SpeedTest

2011-01-28T09:58:00.000-05:00

This is cool.

When you think about it, watching a movie from Netflix or something is a great way to test a network. This is a graph of different big ISPs and their throughput.

In the metric below, we’re filtering for titles that have HD streams available, and for devices capable of playing HD streams (which also filters out mobile networks), to highlight what’s achievable in terms of HD performance on the various ISP networks.

What I find interesting is the high numbers for Comcast. Their light-blue line is consistently #2 on this graph here. I have said that nobody loves Comcast, they either hate Comcast or resign themselves to having to have them, but the data clearly puts them as having a good network.

(Thanks, LifeHacker)

Sikuli X Stress Test

2011-01-24T15:21:00.002-05:00

This video is of a stress test I built for Sikuli, a GUI scripting engine. It uses CV libraries to identify and interact with visual elements on your desktop. I've talked about it, but I haven't shown it, as I haven't had A means to. I'm now running Windows 7 on a fast computer, and I have Microsoft Expression Encoder installed. So, likely I'll have more screenvids in the future.

As you'll see in the code, I'm writing in modules. The first code I show is meant to provide functions for the second code I show.

checking_test


from sikuli.Sikuli import *

def clickAllLikeThis( image ):
 for i in findAll(image):
  click(i)

def checkAll():
 a_unchecked = findAll(unchecked) 
 for u in a_unchecked:
  click(u)

def uncheckAll():
 a_checked = findAll(checked)
 for c in a_checked:
  click(c)

module_test


# works on all platforms
import sys
import os
# get the directory containing your running .sikuli
myPath = os.path.dirname(getBundlePath())
if not myPath in sys.path: sys.path.append(myPath)
import checking_test
unchecked = "1295894889335.png"
checked = "1295894985817.png"
popup( "Checking all unchecked" )
checking_test.clickAllLikeThis( unchecked )
popup("Unchecking all Checked")
checking_test.clickAllLikeThis( checked )
#clickAllLikeThis( checkbox )

The PNG strings are converted in the Sikuli editor, which has screengrab capabilities and converts the file name to the image.

I'm very glad that I'm able to separate functions into separate modules. I'm somewhat disturbed that I'm not getting 100% coverage from the elements I'm trying to get clicked. But it's a neat thing nonetheless.

Java Interface issue that makes no sense to me

2011-01-21T10:08:00.001-05:00

There's a Java tool I have to use, called Blast2GO. It's a Genomics thing and I'm very sure that program itself is not the problem, but it is the only reason I'm using Java. We run this on data wholesale, so we're waiting with bated breath for them to release a command-line interface or API for it, but until then, there's the GUI. And if you're repeating a GUI interface a couple dozen times in a row (easily), it becomes more and more likely that you'll forget something.

So, we're trying to automate it. We're using Sikuli. In a way, Sikuli is a wonder. It uses computer vision libraries to find and click things in a GUI environment. It has allowed us to automate much of the interface, which is a wonder.

In another way, it's totally wrong. GUIs are not a pipeable environment, and if you're going to automate something, command-line scripting is the simplest way. Stupid interface issues pop up rarely that way, while my wrangling with Sikuli is all about handling stupid interface issues.

And the most stupid, the most unsolvable stupid interface issue is this: I run Blast2GO on a Solaris server because that's where the data lives. So, I use X Windows to redirect the output onto my local machine. And right now, wherever I run it from, I lose the ability to navigate menus via arrow keys and move through open and save windows via tab. And, for the life of me, I have no idea why. But whenever I run it on those servers, I can't use those keys. This makes the scripting much harder.

Any idea why Java would be doing this to me? Is it a Solaris issue? A Java issue? An X Windows issue? Help!

Graphical Programming with Yahoo Pipes

2011-01-14T15:02:00.000-05:00

Let us start with the need. I found a tool to allow me to scroll RSS feeds across my desktop, just like every news channel. This is a neat thing, but I found that I didn't like the way it did RSS feeds. Instead of cycling through them, it kept using one. So, I set about solving that by combining feeds.

The method I chose was Yahoo Pipes. When I decided to check Craigslist for items of certain types (Fender guitars, especially) in local areas (where I decided I could reasonably drive, should the catch of a lifetime come around), I used Pipes, mostly as an excuse to learn it. With Google Reader being the recipient of those feeds, I didn't need to massage them too much. The news I fed News was twitter feeds from news sites I wanted to follow, mostly local news and weather.

To the right, we see the "code" I used to combine seven news sources, (mostly) remove redundancies, do some slight editing to the content and ensure that only the day's news shows up. It's all Twitter RSS, so there's one date format, but that format is not usably sortable. I format today's date in the same format and filter out if it isn't there, but the better solution is to take a Unix timestamp of the current time and date (representing seconds since Jan. 1, 1970), subtract 24*24*60 (number of seconds in a day) and drop out entries that were not published during that time. That will take a slightly larger amount of cleverness to do that, but I think I'll crack that nut eventually.

The thing I notice is, while a graphical solution seems tailor-made to mimic flowcharts, there's no splitting here. It's all very linear, with minor loops and filters. I suppose I could get a more funnel-looking program if I handled the inputs separately. The more I play with Yahoo Pipes, the more I like it.

Quick Notes: The Doomed Dozen, The New Radio

2011-01-13T12:39:00.001-05:00

MaximumPC has list of 12 Technologies on the Bring of Disappearing. This is a bit more long-view than I normally go, but it makes sense.

The Doomed Half-Dozen

Pre-recorded Physical Media
Stereoscopic (with glasses) 3D TVs
eBook Readers
Consumer-Level Hard Drives
Keys
Handheld Gaming Consoles

The Half-Dozen Survivors

Digital Music/Media Players
Landline telephone
Internal Combustion Automobile Engine
The PC
The Keyboard
The Mouse

I do have some quibbles with the list, and some more quibbles with the presentation of the list. For instance, if you're going to keep the landline phone alive as a bulwark against power-outages, then why show a portable phone, whose base unit will die as soon as the power is cut? And the eBook reader holds a niche in that the screen makes it a low-energy device, opposed to the power-hungry notebooks, tablets and smartphones. They're mostly single-taskers, to borrow an Alton Brown term, but they can grow up and out the same way the digital media player is. The list is well-worth reading, though.

So, our car is in the shop, and the insurance company has us in a rental. It's a swank piece of motor art, with gratis satellite radio. We're not too used to it, but we're getting there. The Old Radio has been dead to me for quite some time, as it struck me that there wasn't anything I wasn't sick of being sent to me over those airwaves. The little I hear on the XM/Sirius channels I get over DishTV has been interesting, but never interesting enough for me to even consider it for myself. I have a large enough MP3 collection that I could go some months without repeating a song. (That I do repeat songs has something to do with being a musician, with the limits of Rhythmbox's shuffle function, and with the limits of the SD card in my phone.) And, quite honestly, I don't spend enough time on the road to make it worthwhile.

But I'm beginning to wonder about my wife. She does spend great amounts of time on the road, and given her preference for talk radio, I'm thinking it might be good fit for her. Her stations come out of Chicago and are mostly static when she comes to pick me up. But then, I'm wondering if streaming it with 3G or 4G smartphones would be a better solution.

What are your thoughts about dying technologies and/or the New Radio?

Defining Terms: Copy-and-Paste vs Cargo-Cult

2011-01-04T13:59:00.000-05:00

During WWII, in the South Pacific, major sea battles left many sinking cargo ships, whose cargo then floated to neighboring islands. Sufficiently advanced technology is indistinguishable from magic, and the inhabitants of these islands had been barely touched by the events of the 20th century so far, so they took the cargo as signs of gods who should be worshipped. Thus a cargo cult programmer is a person who takes the sample code as the work of a diety, not to be understood and changed but to sit, inviolate, in the code base.

In contrast, copy-and-paste programming is taking a solution for one case and copying it, pasting it, making a small change for the next case, and so on. If there's one line of change in a 20-line copied chunk, this means that there are 19 lines that are duplicated, and if there's a problem, 19 lines that have to be serially debugged and fixed wherever you left them, and you might easily forget many of them. It's better to pull as much as you can to a subroutine so if there's a bug, there is one and only one place you have to go to fix it.

Clearly, two anti-patterns you should try to avoid in your programming life, to be sure, but if you copy code from a website and paste it into your own program, you are not a copy-and-paste programmer. Please respect the distinction.

The New Telephony meets the New Television: Expensive Remote

2010-12-09T00:23:00.000-05:00

I have fairly close to the top end of the smartphones, the HTC Evo Android phone. I have a Vista computer connected to the Internet. I have just installed Boxee on said TV. Thing is, this TV, while it has a big screen and all, is across the room from me, and the keyboard and mouse are corded.

I did say I have an Android phone, right?

While it is a wonder of technology that can talk to WiFi, Bluetooth and 3G/4G mobile networks, and do much cooler stuff. I can watch video. I can make video. It is a cool thing. Being able to control my other computer is minor, but still cool.

I've done something like this before. Years ago, I had a Palm III. You could install an app that made your Palm into a programmable universal remote. That's cool, but that's coolness relying on IR and line-of-site and the IR source was meant for sending electronic business cards 3 feet away, not turning on a DVD player 10 feet away. This works over wireless networking, so line of sight is done, and thanks for that.

I'm also new to virtualizing interfaces. I've been a big fan of Synergy2 to allow you to have one keyboard for two or more computers. I just didn't expect it to touch the smartphone.

I know I'm at best second or third wave on this sort of thing. MythTV's been around for over 5 years, and many of these concepts are very stable. Microsoft's been putting these capabilities as part of their advertisements these days. And I know there are things involved here that I'm taking the goofy way around with. Big example is moving media onto my Android. (I call it RoyBatty, by the way. Considered going with RickDeckard, but no.) The old reliable way to do it would be to plug in the USB/charging cable. The slightly cooler way is via Bluetooth. My way is to use Dropbox. It's a cool and wonderful cross-platform thing. I was expecting full syncing, but it turns out to be pick-and-choose, which is okay, but it means I'm more active.) I think the really cool solution would be to using uPnP/DLNA. This means I have to start learning how to make that stuff work.

But seriously, this really convinces me that the New Television is a big monitor, receiving signal from whatever device you want, and not really the interface device in and of itself.

Now, if you excuse me, I'll watch some Revision3 on my computer as I go to bed.

Stupid HTML Error

2010-12-06T11:41:00.002-05:00

Code Block A.


<form>
    <table id="target">
        <tr><td> <input type="text" name="foo"></dr></tr>
    </table>
</form>

Code Block B.


<table id="target">
    <form>
        <tr><td><input type="text" name="foo"></dr></tr>
    </form>
</table>

Not much difference, is there? In fact, I'd say they're functionally identical, unless you set form as a block in your CSS. But there is a difference if you're going to use Javascript to add to the form. If you append to the table, you'll get something like this.


<table id="target">
    <form>
        <tr><td> <input type="text" name="foo"></dr></tr>
    </form>
    <tr><td> <input type="text" name="bar"></dr></tr>
</table>

And this will not be part of the form when you press submit. Which I have conveniently left out of the code blocks. Hrmm. Still, watch for that. It's the problem that hounded me over the weekend.

First Real Day With Android/EVO

2010-11-29T17:50:00.000-05:00

I received the thing Friday and poked at it all weekend. I don't think I'll have to decide to love this thing.

But I have found a problem. Or at least a non-solution.

I work in a sub-basement. This means no cell connection. I mean, if I sat six feet to the south, there's a slight chance I could get a quarter-bar just enough for a ring. But I don't. So, I have been dreaming of ways to get my phone on. My way has been Google Voice and Gizmo. This meant that I cart my netbook with me. I had been hoping that I could answer calls via WiFi, but that seems to be a no-go.

There's also a Voyage app, but it only works on GSM phones, not CDMA phones like mine. What might work is Skype, but that strikes me as a mighty big if.

So, we'll see how things go.

Have Mercy! A RouteShout Signage Bug

2010-11-17T10:17:00.000-05:00

I now live where the buses go, and I've been taking a bus to work for a little over a month. I can't say there haven't been problems, but I can and must admit that they've all been self-inflicted.

They have signs under all the bus stops saying "Text this code to 25252 and find out how long the wait is". I have tried it, and it fails.

Turns out, this is not an in-house thing, but a service provided through a company called RouteShout. There's more than just texting, though. There's an iPhone app, an Android app, a mobile site for those with older, less-smart phones and an API. That's the point where I said "cool" and dove in. And found the problem.

Say you have a bus stop. Say CityBus labelled it BUS666. Well, if you're gonna get on a bus someplace, you'll want to come back near there, too. So, we can think of BUS666 as where you get picked up and where you get let off. Except, sometimes the bus will go up and down the same road. For example, the bus I take from my apartment goes around a mall and comes back. So BUS666 is on both sides of the street. If I'm waiting on the north side of the street, I don't care about buses coming to the south side, except when they're coming around and back to the north side. So, we have BUS666N and BUS666S.

But the signs still say BUS666.

And if you text BUS666 to 25252, rather than BUS666N. Which makes the texting interface useless. And you have to get inside the API to figure out where the problem is.

(As it turns out, BUS666 is one stop, not two, and doesn't have this problem. It also is far away from anywhere I take the bus to, so using it as an example doesn't lead to people knowing where I live.)

Again, this is a signage bug. The RouteShout service works perfectly, as far as I've seen. I think I'll put together a Perl module from the API and put that on CPAN. Yay to them.

Solving a Brain Teaser with word lists and Perl

2010-11-10T12:05:00.000-05:00

I'm a programmer. When I see a problem, I get tempted to code the solution. One place where this temptation comes into play is brain teaser problems. Once, a while ago, a radio show had a brain teaster where you take the phrase PRECHRISTMAS SALE, turn it into a 4x4 block and find the longest word you can without reusing a letter. I used Perl and found six eight-letter words.

And, of course I used my powers on Sudoku.

Now, I've seen a problem that's inspired me again.

By changing one letter at a time to form different English words, and leaving all other letters in their original positions, convert SIXTH into TENTH in the fewest steps possible. Good luck!
S I X T H
_ _ _ _ _
_ _ _ _ _
_ _ _ _ _
_ _ _ _ _
_ _ _ _ _
_ _ _ _ _
T E N T H

There are Spoilers below

As I learned with PRECHRISTMASSALE, which I've since found is Boggle, if you're going to find words, you first need to know words. So, I pulled and adapted my Boggle code for coming up with a dictionary list.

sub dictionary {
    my ( $length ) = @_ ;
    my %done ;
    my %word ;
    my $dir = './Dict' ;
    my $dirhandle ;
    opendir $dirhandle, $dir ;
    my $file ;
    while ( defined( $file = readdir $dirhandle ) ) {
        next if $file =~ m(^\.)mx ;
        open my $filehandle, '<', $dir . q{/} . $file ;
        while ( <$filehandle> ) {
            chomp ;
            $_ = lc $_ ;
            next if $done{ $_ }++ ;
            next if length != $length ;
            $word{ $_ }++ ;
            }
        close $filehandle ;
        }
    closedir $dirhandle ;
    return sort { $a cmp $b } keys %word ;
    }

Of course, you need to be in the same directory as a directory full of dictionaries. I used a few I pulled from the CERIAS FTP site, and probably has some overlap, but better that then missing a word and thus a connection.

I then started poking at the words, trying to find the next. I came up with a way to get all the proper alternatives, given a word. I then pulled it into a subroutine.

sub get_word_choices {
    my ( $word ) = shift ;
    my @word = split m{}mx, $word ;
    my @output ;
    for my $pos ( 0 .. scalar @word - 1 ) {
        my @local = @word ;
        $local[ $pos ] = '.' ;
        my $local = join '', @local ;
        my @words = grep { m{$local}mx } grep { !m{$word}mx } @dictionary ;
        next if scalar @words == 0 ;
        push @output , @words ;
        }
    return @output ;
    }

I should probably pass @dictionary rather than leave it as a global, but certainly, you don't want to re-parse the dictionary file each time.

It struck me about this point that I should have all I need to automatically solve this, not leaving me to use this tool to piece something together.

I am not as happy with this part as I could be.

sub check_words {
    my $word2 = shift ;
    my @words = @_ ;
    return if scalar @words > 7 ;
    my $last = $words[-1] ;
    if ( $last eq $word2 ) {
        say '' ;
        say join ' ' , @words , scalar @words ;
        }
    my @choices = get_word_choices( $last ) ;
    #say join ' ' , @words ;
    for my $choice ( @choices ) {
        next if grep { m{$choice}mx } @words ;
        check_words( $word2 , @words , $choice ) ;
        }
    print join ' ' ,  '' , scalar @words ;
    return ;
    }

It's depth-first search and it's long. Which, I suppose, is unavoidable for depth-first search. Consider this solution:

sixth
sixte
sixty
silty
tilty
tinty
tenty
tenth

sixth to sixty is just as legal a jump as sixth to sixte. If I used an iterative process with a better data structure, I could use a shortest path algorithm and bypass sixte. And, of course, by putting print to give me some appreciation of progress, I'm bogging the thing way down. So, that's where I'd rewrite to make this better.

Somehow The New Television is listening to Me

2010-11-08T10:44:00.000-05:00

And not in the creepy HAL2000 kind of way.

I was searching for info on Net Neutrality, wrapping my head around it, and found this bit from G4's Attack of the Show on Hulu. Next clip on the channel was on Bluetooth headsets, and it was presented by Dungeons and Dragons. Which was a bit of a wow for me.

I've been clear on this, but let me restate that I am not categorically against advertising because advertising, properly done, is content. Consider the copy of The Productive Programmer on my desk right now. The last page before the back cover is an ad that says you can read this book online for 45 days at Safari Online. If you're reading a book called The Productive Programmer, you're more than likely a programmer. Which means you deal with computers a lot. Reading this book online will make sense to you, as you read much online. It's as much content to you as the book's suggestions for Desktop shortcuts. This doesn't mean that you, as the programmer reading this book, will jump up and start reading online any more than it means that you will immediately pick up Virtual Desktop Manager to get virtual desktops on your XP dev box. But it'll be something at least interesting to you.

However, if the back-of-the-book ad was Chanel #5 or Audis or Levi's jeans, it would be far less clear that there's a connection between the audience and the advertisement. This is where mass-market media starts to fall apart. They make a statistical model of their audience, or who they want their audience to be, and advertisers who want to sell to that audience buy ads. If you are not in that statistical model, or the statistical model is too general, the ads will not speak to you.

I haven't been an RPG gamer in two decades. D&D is not going to sell me, but it seems a more natural fit to the G4 audience than cars.

The New Television is similar to the (Older) New Telephone

2010-11-05T15:14:00.000-04:00

When I was in college about a decade ago, I knew some people who didn't have a "landline", or a traditional wired telephone in their place of residence. It made sense to me for college students. If you're out and studying, going to classes, working between classes, etc., it made sense that you weren't home enough to hear your phone ringing, so it's best to just have the mobile phone and not worry about it. For students, sure that made perfect sense. For adults, however, I wasn't sure. Come ten years after, you have more and more people who are dropping the landline that are out of school. There are sticking points, like how E911 for mobile isn't quite as far as for traditional telephony*, but it's a trade-off more and more making.

I have an adapter for my old-school TV that allows me to plug my netbook's VGA and watch YouTube, or whatever, on the big screen. My Wii can do Netflix, as can my friend's iPhone. Android is coming soon. Hulu can bring most of my favorite shows to me. Not everything, but close and closer all the time. If this is true, if you can get all the moving pictures you need from the internets and don't need the co-ax run. And we're not talking people who can't afford cable. We're talking people who decided that they just didn't need another wire run and time-sink in their lives.

The comments highlight a few of the hinderances with this. First, if you want to watch live events, like sporting events or news, you're out of luck. If there's a way to see Big Bang Theory or The Marty Stuart Show or new episodes of Discovery shows online, I've yet to see it. And this model is more-or-less fine if you know what you watch and watch what you know, but it locks you out of searching through and finding the next cool thing you didn't know existed. So, TV-wise, I'm not a cord-cutter yet. But I'm moving that way.

* The two times I've had to dial 911, I was calling because of a car accident.