He hates these tabs! Stay away from the tabs!

My preferred KomodoEdit settings. 4 spaces per indent, tabs display as ZERO characters. Don't even show the things. Certainly don't make the Tab key put one in.

My response to editor developers who make my editor add tabs.

Networking Fun

What we have here are two browsers running of a local machine (XP, but that's not important right now), and one X-forwarded from my home, all trying to get to perl.org.

At work, I cannot go to perl.org or cpan.org. At home, I can.

If I can route to home, and home can route to perl.org, I should be able to route to perl.org. And I don't know why I can't.

Terry Childs

The case is in the hands of the jury. I might have missed something, but I don't think I heard of a single action he took that wasn't by-the-book protecting the network that the first-responders of San Francisco use to protect their city. San Francisco jails system adminstrators.

Free Terry

Sour Milk

A co-worker has an HP xw8000 for a workstation. It runs XP, and if you leave XP going, it starts getting slower and such, so she rebooted it.

And it wouldn't start.



That's the beep she got. This is where I came in. I moved it, opened the case, unplugged it, made sure all the cards were in tight — the last time there was a problem with this box, it was a loose video card — and began to check the beep code index.

There's no entry for no beeps.

What am I supposed to do for no beeps? "Hey, guys? I have a problem, so I won't start. No, I won't tell you what the problem is."

So, I told the other tech guy.

He unplugged it and waited until the green light on the power supply went out. I am sure that happened in my picking up and moving, too. Then he plugged it in and started it.

And it started.

The co-worker wanted to know what the problem was, what she did to upset the silicon god on her desktop.

For the life of me, I can't understand what the problem was, much less explain it to someone else. Although, I think I have a start.

The Return of jQuery Madness: SQL Re-Engineering

In response to my previous jQuery Madness post, Leonard said

I had a similar problem with ajax calls. I had jack dropdowns: building, Floor, room, jack. One would populate the next and then the next etc, just like yours, except I didn't automatically pick the top selection.

When making multiple calls to any API/framework/database I always ask myselfr: What do you know when calling the second call, that you didn't know with the first call. The user hasn't put in more data, so is there a valid reason for more load on the other end of the pipe? Could you have done all the work with the first call?

I would suggest re-engineering your calls just a little bit so that when foo is changed, your backend assumes that you will want the first bar/blee/quuz. (similar assumptions for bar and blee).

This way you only need one ajax call per selection onChange.

This seems logical. Remember that at one point, AJAX meant Asynchronous Javascript And XML, and while lots of people have rejected the XML in favor of JSON, RSS, or plain text, making it AJA (see also: Steely Dan). But until this, I don't know that I had really considered the asynchronous part.

As an experiment, I've created a page that loads everything in one page, logging the URLs at the bottom of the page with .ajaxComplete(), and you can really see everything come together in it's own order, not the order I want it to.

I now reveal that foo, bar, blee and bar are really lab_director, request_id, accession_id and barcode. Each lab_director has one or more request_id, each request_id has one or more accession_id, and each accession_id has zero or more barcode. null counts as a barcode. Here's the code I have so far.

SELECT  re.lab_director , re.request_id  , acc.accession_id , bc.barcode
    FROM    requests re     , accessions acc , accession_barcode bc
    WHERE   re.request_id    = acc.request_id
        AND acc.accession_id = bc.accession_id
    ORDER BY lab_director

The problem is, this only works for things that have a barcode. I could get everything but barcodes and add to the data structure as it comes up, but if I could get something like above, except barcode being filled in as null if there's no barcode for that accession_id in accession_barcode.

Any pointers? I don't even know if I could hammer this down to a googlable question.

Graphical Programming

I've worked with it before, 2 years ago when I was working with a defense contractor. There, they used it as a way to not have to write C++, and since they were electrical engineers and not computer guys, I can get not wanting to write C++. That's the closest I've ever gotten to being paid for it myself. But I had enough problems figuring out some problems with the generated code that I soured on the concept of Graphical Programming.

Until LifeHacker came up with solutions for improving your Google Reader feeds, including Yahoo Pipes, which has it's own Graphical Programming bit.

I've done a few. I've taken the Craigslist instrument feeds and searched out organ and Fender (as Fender guitars and amps and Hammond organs are the things I most want), and pulled out Pimp My PC from Butterscotch, as well as FLOSS Weekly from TWIT.TV.

It's a neat thing. I haven't done much more than touch the most simple part of it, so we'll see how it goes, if I can make it do stranger things. But it's a neat idea.

jQuery Madness

I have a foo.

For every foo, I have one or more bar.

Each bar has one or more blee.

Each blee has one or more quuz.

I can get the list of all foos via AJAX, in convenient JSON form. By passing up my foo, I can get a list of all bars associated with said foo. By passing up my bar, I can get a list of all blees associated with said bar. And so on with quuz. With Perl and DBI and JSON, this is relatively easy to do.

Now, I have a web interface for all this, with select dropdowns. When you choose a foo, all the current bar, blee and quuz selects are emptied and refilled.

I start out with a empty position for foos and blank bars and blees, one with no value. When I choose a foo, it gets the bar and fills that select, with the empty position selected. Well, the boss wants it so, if you choose a foo, the top bar is brought up, the top blee of the top bar is brought up, and the top quuz of the top blee is brought up. Thing is, by the time that we're trying to load blee, quuz might've already been starting.

It's the timing of the deal that I think is the problem.

Welcome (Back) To The Working Week

Spent two days off in abstract-land, attending the CERIAS Security Symposium. That's always a good time, filling my head with mind-expanding thoughts that have no real practical application in my life or career.

My friend Patrick also attended, and as he is a systems guy for a defense contractor, there's slightly more practical application for this in his life. Beyond, you know, security being part of everybody's business. He observed what I began to feel last time, that there wasn't much technical in the presentations and a lot was social and/or organizational, with David Bell of the Bell-La Padula model telling us that everything he needed to know about computer security, he learned in 1974. Which adds to my belief that computer security is a solved problem, but implementing and distributing the solution into the real world is not.

Which, if I recall correctly, is why COAST turned into CERIAS in the first place.

When I got back to the office, a co-worker had an interesting problem.

After a reboot, she could not type a password. Nor could I. With neither the user or admin account. This was a graphical-login XP machine, so there was no means to see what the computer thought I was typing. So, I tried the first thing any IT professional tries. I tried turning it off and on again. And it worked. Still no clue as to what the problem was.

Working through a Module Problem

I have a module. This module is too big to be managed. So I've broken it up. It was Dumb.pm and I've broken it into Dumb/Database.pm, Dumb/HTML.pm, Dumb/This.pm, Dumb/That.pm, and Dumb/TheOtherThing.pm.

Yes, I am convinced of the low quality of this code. Yes, I wrote every dumb line of it myself. No, I will not post it for the amusement of the masses.

Under all this, there's 2 databases. One for test, one for production. If you want to test things, test against data that isn't vital, right? With Dumb.pm, I took something from Data::Dumper, specifically how you set indentation, so that you put in $Dumb::Database = 'test' ; if the you wanted to use the test DB. Usually this is connect to an if statement, like $Dumb::Database = 'test' if defined $cgi->param('test') ; It's a nice, compact solution. But if you balkanize the code base, there's not one bit you can set to say "look to the DB".

My first pass semi-solved this by having hash refs for getting variables into subroutines. Dumb::That::this_function( $hashref ), so as long as $hashref->{database} is set, or the underlying subroutine knows that an unset var means production, not test. But I am requested to make it more like the previous. More like Dumb::That::this_function( $value ).

Let me say that, if you're going to put several dozen variables through, enough so that ensuring order is a major concern, sending hashes or hash refs is a good idea. There are points where I hit that threshold, and I've kept it as the general case, even when I'm just passing one or two variables.

As I can figure it, this means having a reborn Dumb.pm, call it Dumber.pm as a central module, which has the $Database var. Then, every time I want Dumb::This::subroutine(), I go for Dumber::this_subroutine(), which is a simple wrapper that handles setting $hash->{database} and passes it on to Dumb/This.pm.

I see this as a tree issue. Child nodes can see what's in the parent node, but parallel children can't see each other. I don't see how biting the bullet and finally getting into Object Orientation would help, and I'm resistant anyway because that's one more dang thing to learn and kick this back. If there was another obvious solution, I wish somebody would give it to me.

As I wrap this up and resign myself to diving into this, let me mention the usefulness of passing hashes or hash refs if you're moving massive amounts of data into a subroutine. $result = my_subroutine( $a , $b, $c, $d , $e , $f , $g , $h , $j , $i , $k , $l ) will make you lose track of your variables quick, and passing them with a hash means you know what's what. CGI pushed me that way, too. Not useful for everything, but it has points.

Higher Order MP3 Directory Organization, A First Step

I have a huge number of MP3s. I am sure I haven't heard all of them. Some have weird tags. Some have no tags. Some are not really MP3s, but "you can't download this" HTML files, or just zero-sized files. When I bump into them, I can fix these things (read: delete the bad files) but it can take some time.

I had wanted to use the power of Perl to help with this, but while there are great numbers of modules to help with just about anything, I didn't have a directory walker I liked.

Then I started trying to go through Higher Order Perl by Mark Jason Dominus. And one of the first examples is a directory walker which takes anonymous subroutines. Exactly!

First step was to make a script that counts my MP3s.

#!/usr/bin/perl
use 5.010 ;
use strict ;
use warnings ;
use Carp ;
use Data::Dumper ;
use MP3::Tag ;
use MP3::Info ;
use Digest::SHA1 ;
use lib '/home/jacoby/lib' ;
use HOP ':all' ;

my $x   = 1 ;

# dirwalk home directory , file handing sub , directory handling sub
dir_walk(
    '/home/jacoby/Music',
    sub {
        my $file = $_[ 0 ] ;
        return if $file !~ m/mp3$/imx ;
        $x++ ;
        } ,
    sub { },
        ) ;

say $x . ' MP3 files' ;
exit ;

Everything that claims to be an MP3 gets counted. Yay! (Just so you know, the current count is 37604.) There's lots of included modules that I don't use yet. HOP.pm simply puts MJD's directory walker into a module where I can get it on demand, so I don't have to copy and paste. Having a command-line set for the directory would be good, but not today.

And needless to say, you can adjust this to do a lot of other things. Check file sizes. Find file names without track numbers. Stuff like that. There are three downsides so far: You don't have hashes to find repeated songs, you don't have MP3 tag information, and you have to run it again (with the associated lag of running a directory walker on 30,000+ MP3s.

But there are solutions.

Digest::SHA1. MP3::Info and/or MP3::Tag. DBI.

I run Linux. sudo apt-get install mysql-server gets me a DB. Run once, save the data and query until you're sick. I started out with this schema.

CREATE TABLE music (
    id              int(20) NOT NULL auto_increment primary key ,
    album           VARCHAR(255),
    artist          VARCHAR(255),
    filename        VARCHAR(255),
    filepath        VARCHAR(255),
    filesize        int(32),
    length          int(32),
    release_year    VARCHAR(4),
    run_length      VARCHAR(32),
    sha1_hash       VARCHAR(255),
    title           VARCHAR(255)
    ) ;

length is song length in seconds. run_length is song length in HH:MM:SS format, and yeah, I have some MP3s that push that, if not exceed it. Or that's the theory, at least.

And some would say it's bad schema design, but I'm not so much worried about grouping by artist or album or year. Those tell me if the file has ID3 tags or not. I'm focused on the MP3 file itself here.

#!/usr/bin/perl
use 5.010 ;
use strict ;
use warnings ;
use Carp ;
use Data::Dumper ;
use MP3::Tag ;
use MP3::Info ;
use Digest::SHA1 ;
use lib '/home/jacoby/lib' ;
use HOP ':all' ;
use MusicDB 'db_connect' ;

$Data::Dumper::Indent = 1 ;
$MP3::Info::try_harder = 1 ;

my $sql = <<"SQL" ;
INSERT INTO music
    (
    album       , artist , filename     , filepath      ,
    filesize    , length , release_year , run_length    ,
    sha1_hash   , title
    )
    VALUES
    (
    ? , ? , ? , ? ,
    ? , ? , ? , ? ,
    ? , ?
    )
SQL

my $dbh = MusicDB::db_connect() ;
my $sth = $dbh->prepare( $sql ) ;
my $count = 1;

dir_walk( '/home/jacoby/Music' , \&mp3_check, sub { } ) ;
exit ;

sub mp3_check {
    my $file = $_[ 0 ] ;
    return if $file !~ m/mp3$/imx ;
    my $filename = ( split m{/}mx , $file )[-1] ;  # just the file name
    open my $fh, '<', $file or return ;    # for SHA1 HASH
    my $hash = Digest::SHA1->new ;         # for SHA1 HASH
    $hash->addfile( $fh ) ;                # for SHA1 HASH
    my $digest = $hash->hexdigest ;        # for SHA1 HASH
    my $mp3    = MP3::Tag->new( $file ) ;  # for MP3 tags
    my $size   = -s $file ;                # for MP3 tags
    my ($title, $track, $artist, $album,   # for MP3 tags
        $comment, $year, $genre )
      = $mp3->autoinfo() ;                 # for MP3 tags
    my $total_secs = $mp3->total_secs_int() ;
    my $time       = $mp3->time_mm_ss() ;
    $sth->execute(
        $album       ,
        $artist ,
        $filename     ,
        $file ,
        $size    ,
        $time ,
        $year ,
        $total_secs    ,
        $digest ,
        $title
        ) ;
    say $count if $count % 1000 == 0 ; #to keep track of progress
    $count++ ;
    }

This is still a work in progress. I don't use Carp here, but I generally include it when I should. As I'm debugging, I always have Data::Dumper floating around so I can see what the data structures are. I could probably just use MP3::Info instead of MP3::Tag. Haven't decided yet. Digest::SHA1 gives a cryptographically-secure hash of the MP3, so that should detect duplicates. HOP was mentioned earlier, and MusicDB is a wrapper module that allows me to have my DB passwords in one convenient place, so I just have to worry about the actual SQL. There are some bugs — length doesn't give the right info yet — but I have all the info on any discrete MP3 file.

Notice though, that the function has become sufficiently big and complicated that I've pulled it out and given it a name. Also notice how I'm starting to use placeholders, which should make my DB interface more efficient.

A good thing to add would be to see if a file has been put into the DB, and if so, to get the unique index, file size and hash to check for changes, then update only if there's changes, rather than inputting it in again.