Ghosts of Coding Sessions Past

Boss wants to be able to clone a request, to take an item on our list and reset it.

Fair enough, but ...

Requests are among the first things I coded at the lab. Before that, I was at the clinic, and while I did a fair amount of coding there, I was much more of an admin than a developer there, and I have learned a lot about developing here.

Thing is, I learned most of it after I wrote he request code.

A request has lots of data about it. A name, an ID number, who requested it and how to get in contact, what they want done with it, etc. Each request has a number of samples, and the samples have names and numbers and amounts and such. Each sample can have a barcode, which I won't explain. Each request can also have 0-whatever statuses. I back-end to MySQL, and I can copy and massage data, and say in the status that request j is a clone of request i. All that is cool.

Then there's the Wiki.

About everything on the web page is also on the Wiki. It serves as the lab notebook, and I create the stubs. Yes, there's synchronization problems between the Wiki and the database. I try not to think about it, because I cannot find a way for the structured data to interact with the unstructured in a good way.

So, I dread the Wiki stub creation code. I coded it in Perl, using the CGI module. Before I came to the lab, I hand-coded my HTML in my Perl, like my $val = "<div> $foo </div>" which isn't good either, but 200 lines of code like $val = $cgi->div( $foo ) isn't doing my mind any good.

Thrown into the mix, we're moving from our old server to virtual server Sidam, which is Midas spelled backwards, as everything it touches turns to sh*t. I should be moving my code to that server, but that looks to be a bigger issue. To make sure that things finish, we want to move the code as close to as-is as possible. Because it's broken by design (see above) I want to use this opportunity to make great, sweeping changes.

I think I'll implement this with Template Toolkit and suck up the technical debt, but I don't look forward to that process.

How a technical presentation Q&A session turned my head around.

Sometimes you learn something beyond what you expect.

Last night, I attended a presentation at my local LUG of Puppet by Garrett Honeycutt of Puppetlabs. Garrett had evidently been a PLUG guy when I was, but I didn't really recognize him. I did recognize his friend, so it could be my mind going.

Anyway, Puppet, in a nutshell, is a configuration manager useful for keeping hundreds of machines doing what you want, using the same kinds of tools people use to manage software, including moving from Development to Test to Production. Some people I know who ride the big iron are finding their current means to manage their hundreds of machines to be a little too hodge-podge and daunting were there, and it seems they like the taste of that Kool-Aid. I was interested, and thinking it could be useful, but a bit more than anything I need.

That's when it hit me. In our current workplace, we're not really doing that. There's one canonical instance of a thing, be it a Perl program, a Perl library, a Javascript module, a web page, a script, whatever. We have test and production databases, and for some of the modules, we're using tests like in Perl Testing, which I must confess I never really "got". So, while we might be generating decent code, we're not engineering it and maintaining it the way we should.

So, I'm spending much of today trying to wrap my head around that, hoping to establish Dev-Test-Production setup for my code. Thank you, Garrett and Mike and the rest, for kicking me the right direction without even meaning to.

Failure to Plan is Planning to Fail

I have a database table full of requests. When that table is filled, we also put together a wiki page for that request, because the the data we generate for each request can be fairly free-form. This means that we effectively divorce the state of the DB for a given request and the state of the wiki.

There are certain fields that are on the wiki page and not the database. Specifically, contact information for the person making the request. It made sense to me, at the time. I think it was based upon the data separation issue, or that we actually hold that in a profile, too, but I can't remember right now.

Now, my boss wants to be able to regenerate the wiki page for quick-and-easy refresh. Regenerate the DB stuff into a new wiki page and cut-and-paste the non-DB stuff from the old page to the new page. Easy-peasy. Except, I don't store all the data in the database.

So, now, I'm looking at adding columns to the table to handle said contact information and spidering the contact info off the wiki pages. If I had just put it into the database in the first place...

Java Interface issue that makes no sense to me

There's a Java tool I have to use, called Blast2GO. It's a Genomics thing and I'm very sure that program itself is not the problem, but it is the only reason I'm using Java. We run this on data wholesale, so we're waiting with bated breath for them to release a command-line interface or API for it, but until then, there's the GUI. And if you're repeating a GUI interface a couple dozen times in a row (easily), it becomes more and more likely that you'll forget something.

So, we're trying to automate it. We're using Sikuli. In a way, Sikuli is a wonder. It uses computer vision libraries to find and click things in a GUI environment. It has allowed us to automate much of the interface, which is a wonder.

In another way, it's totally wrong. GUIs are not a pipeable environment, and if you're going to automate something, command-line scripting is the simplest way. Stupid interface issues pop up rarely that way, while my wrangling with Sikuli is all about handling stupid interface issues.

And the most stupid, the most unsolvable stupid interface issue is this: I run Blast2GO on a Solaris server because that's where the data lives. So, I use X Windows to redirect the output onto my local machine. And right now, wherever I run it from, I lose the ability to navigate menus via arrow keys and move through open and save windows via tab. And, for the life of me, I have no idea why. But whenever I run it on those servers, I can't use those keys. This makes the scripting much harder.

Any idea why Java would be doing this to me? Is it a Solaris issue? A Java issue? An X Windows issue? Help!

Sour Milk

A co-worker has an HP xw8000 for a workstation. It runs XP, and if you leave XP going, it starts getting slower and such, so she rebooted it.

And it wouldn't start.



That's the beep she got. This is where I came in. I moved it, opened the case, unplugged it, made sure all the cards were in tight — the last time there was a problem with this box, it was a loose video card — and began to check the beep code index.

There's no entry for no beeps.

What am I supposed to do for no beeps? "Hey, guys? I have a problem, so I won't start. No, I won't tell you what the problem is."

So, I told the other tech guy.

He unplugged it and waited until the green light on the power supply went out. I am sure that happened in my picking up and moving, too. Then he plugged it in and started it.

And it started.

The co-worker wanted to know what the problem was, what she did to upset the silicon god on her desktop.

For the life of me, I can't understand what the problem was, much less explain it to someone else. Although, I think I have a start.

There's Something To This

Ed Finkler (aka @funkatron and the developer of the AIR-based Twitter client Spaz) wrote on the subject of the user experience.

These people have better things to do with their days than tweaking out the spacing in their browser toolbars. A computer for them is a utility. One that is increasingly complex, and one that is used because it’s the only option for accomplishing certain things – not because it’s a good option.

It’s kind of like the Photoshop Problem: when people want to crop a picture, we give them Photoshop. Photoshop is a behemoth application with nearly every image editing and touchup function imaginable, and it is terribly complex. Now Photoshop is an impressive tool, but only a very tiny percentage people need the power it offers. The vast majority just want to crop their ex-husband from the photo and let their friends look at it. But even iPhoto, the poster child for Apps So Easy Your Grandparents Can Use Them, continues to pile on features and complexity.

When folks need an elevator, we should give them an elevator, not an airplane. We’ve been giving them airplanes for 30 years, and then laughing at them for being too stupid to fly them right.

I think we’re the stupid ones.

Forgive me if I am wrong, but I think there's crossover here with Jeff Atwood (@codinghorror) and his comments on netbooks.

They may be pieces of junk to Mr. Jobs, but to me, these modest little boxes are marvels -- inspiring evidence of the inexorable march of powerful, open computing technology to everyman and everywhere.

We have produced They may be pieces of junk to Mr. Jobs, but to me, these modest little boxes are marvels -- inspiring evidence of the inexorable march of powerful, open computing technology to everyman and everywhere.

We have produced a democracy of netbooks. And the geek in me can't wait to see what happens next.

There are a couple moves I can make from here. Don't know which is the best. So I will poke at it and try to find the right angle. Part of that angle is that the desktop PC is a dinosaur that's feeling the cold coming on. When I moved into my house, I ran CAT5 to the master bedroom. I crawled into the crawl space. I drilled holes through the floor and cut holes in the drywall. I bought a 2-foot long drill bit. I have not used that line in years. I have WiFi. I know the problems with WiFi, and how this means my neighbor can mess up my WiFi if his config is the same as mine (yeah, it happened) and I know that no wireless network will be as secure as a wire, but I don't care, because the win exceeds the lose.

My work is in a lab at a university. We have several instruments, tools to do science. Many of them are so complex and have computers to control them, and sometimes (not always) collect data from them. Dealing with these machines is a major part of my work day. One of the newer instruments came with a laptop. I groused, saying that this was small enough that someone could grab it, shove it in a bag and walk away, but it was put there. There's a lot to having a full-tower desktop PC that makes a whole lot of sense if you're going to push the full-power of the PC. If you expect to swap hard drives like a guitarist swaps strings, if you plan to add or shuffle PCI cards like a Vegas dealer, the full-tower case makes sense. If you're going to leave it sit, do it's business and whir, a laptop, or even a netbook, makes sense.

Similarly, if you're going to "compute", which in the modern sense means to browse the web, check mail, chat with friends and the like, wherever you might be, the end-all be-all of computer design circa 1999 on steroids is not going to be what you need. Weight, networking and power requirements lock you to one spot with desktop PCs, but are complete non-issues with netbooks. I have a reasonably smart phone. I have had an appreciably smarter phone. I don't walk around with the current best-of-class smartest phones (iPhone or Android). The one I have, a Samsung Instinct, is smart enough for much of what I do. It plays music. It plays video. It talks to my Google Calendar. It sends and receives mail. It isn't all the "computing" you will ever need, but it is a pretty useful subset. When my wife's netbook (and primary computer) went back to Asus for a repair, she relied on her phone and was able to do most of the stuff she needed to do.

I wouldn't want to develop on a netbook, and when I can, I use Synergy2 or the like to allow me to use a normal keyboard and mouse with mine. But I'm fine for tweeting and blogging and music and IM and such on it.

Years ago, I saw an article that compared and contrasted the workstation of 1990 and 2000. (I think. I can't find it.) You had a huge difference between the capabilities of the hardware ( 10baseT vs 100baseT, huge differences the amount of memory available, much larger drive sizes and processor speeds ) and stagnation in the
software ( X and a window manager holding a terminal window, into which you type via EMACS or VI and compile with gcc or the like ). If you do a similar comparison between the platforms of the common user vs a developer's setup, you'll get a big delta in both the hardware and the software. The computer the developer might be much faster than it was before, but it looks like a box with a mouse and a keyboard and a monitor, with a power cord and a network cord giving up to gigabit ethernet. The computer the end-user might look like that. It might have the HDTV as a monitor and a remote to serve as the mouse. It might be a netbook smaller than external CDROMs were five years ago. It might be a phone.

I flash back to the oil embargo of the early 1970s. This is where the definition of car that the US auto makers and the consumers diverged, leaving an opening for the Japanese car companies. I think it is clear that, hardware-wise, the industry is keeping up with demand, making smaller, lighter things with cupholders, but people used to having computers with two-tone paint and fins and big V8 engines (in this case, Steve Jobs and lots of programmers) are making the software like they used to. Like makes sense for them. Which is a great disconnect, which is where Ed's comment comes in. I think.

More later, when I can re-think this and force it to make more sense.

Red Octagons saying "Go!"

Remember when Windows used to make you click Start to stop the machine?



Now you click Shut Down with no option to shut it down!

Self-Introduced Bugs

So, I have 3 different SQL statements that handle three different cases, and the mechanism that dealt with it spat out hashes. (I code Perl. Just making this clear early on.)

I wanted to unify the three hashes into one hash. The way I liked best was to push pointers to the hashes into an array, iterating through that array, and taking each hash and dumping the values into a master hash.

    my %te_ids ;
    my @loop ;
    push @loop, \%te_ids1 ;
    push @loop, \%te_ids2 ;
    push @loop, \%te_ids3 ;
    for my $hash ( @loop ) {
        for my $k ( keys %te_ids2 ) {
            $te_ids{ $k } = 1 ;
            }
        }

That might not be the best way to join hashes, but it is the way I liked. I have the join code once, so if I decide "hey, this works much better!", I don't have to fix it more than one place.

But do you see the problem? Do you see why I wasn't getting values off %te_ids1 and %te_ids3? Do you see it?

I didn't. Not for a while.

(And to be honest, I used sort keys %te_ids2, when sorting the keys added to the overhead with no benefit at all. Just a piece of cargo-cult programming I have to rid my mind of.)

This is the fixed version. The version where I actually use the refs to hashes I shove into the array.

    my %te_ids ;
    my @loop ;
    push @loop, \%te_ids1 ;
    push @loop, \%te_ids2 ;
    push @loop, \%te_ids3 ;
    for my $hash ( @loop ) {
        for my $k ( keys %$hash ) {
            $te_ids{ $k } = 1 ;
            }
        }

Exercising futility ...

Run within R, it works. Run from test_R.pl, it doesn't. Why?

jacoby@oz:~/Desktop$ cat test_R.pl 
#!/usr/bin/perl

use 5.010 ;
use strict ;
use warnings ;
use Carp ;
use Cwd ;
use File::Copy ;
use Statistics::R ;

my $cwd = getcwd ;

my $R = Statistics::R->new( ) ;
say 'starting R' ;
$R->startR ;

while (  ) {
    chomp ;
    my $line = $_ ;
    next if $line =~ m{^q}mx ;
    say $line ;
    $R->send( $line ) ;
    sleep 1;
    }

$R->stopR() ;
say 'stopped R' ;


__DATA__
png(file="~/Desktop/plot.png")
plot(pressure)
postscript(file="~/Desktop/plot.ps")
plot(pressure)
jpeg(file="~/Desktop/plot.jpg")
plot(pressure)
q("no")
jacoby@oz:~/Desktop$ R

R version 2.7.1 (2008-06-23)
Copyright (C) 2008 The R Foundation for Statistical Computing
ISBN 3-900051-07-0

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> 
> png(file="~/Desktop/plot.png")
> plot(pressure)
> postscript(file="~/Desktop/plot.ps")
> plot(pressure)
> jpeg(file="~/Desktop/plot.jpg")
> plot(pressure)
> q("no")
jacoby@oz:~/Desktop$ ./test_R.pl 
starting R
png(file="~/Desktop/plot.png")
plot(pressure)
postscript(file="~/Desktop/plot.ps")
plot(pressure)
jpeg(file="~/Desktop/plot.jpg")
plot(pressure)
stopped R
jacoby@oz:~/Desktop$ ls plot.*
plot.jpg  plot.png  plot.ps
jacoby@oz:~/Desktop$ twitter.pl "It works from R but not via Statistics::R. It does not make sense."
It works from R but not via Statistics::R. It does not make sense.
OK
jacoby@oz:~/Desktop$ 

I Am Curious (Yellow) ...

Or, rather, my monitor is.

Yellow, that is.

My Win7-running test PC has an AGP slot. I have never run anything in the AGP slot. I now remember why. Because it sucks. I'm running much better than 640x480. I'm fully capable of hitting 1600x1200. But in yellow.

Which is my hardware's fault, not Windows 7.

I'm thinking about a statement I once made. You don't need 3D acceleration to run a window manager. Yet.

It said it wants hardware acceleration to bring out all the glory and wonder that is the game Minesweeper. And again with Mahjong. Ye gods.

Well, my hardware is old and crappy. I know this. But I think that, when we start considering 3D graphics for the window manager, I should stand athwart yelling stop.

Until, of course, I get better hardware and get Compiz running.

Also, should we start calling it Windows System Seven?

Aww. Why not?

Grading the Win7 Beta...

I like the fish.

The UI looks OK.

Or at least it would. I can't better than 640x480x16bit, and nothing looks good at 640x480x16bit.

They have integrated "Clear the screen" into the alt-tab, which is cool. But small fonts rendered big look bad. It networks, it seems OK, I might trust it on my family PC, but seriously. I've rarely had a piece of hardware not recognized by Linux installers riding on CDROMs. I've gotten 1600x1200 on this monitor, and now I'm getting 640x480 and it came on a DVD. WTF?

Restating the theme, which is "We Live In The Future"...

Rage Taking Over ....

The current job is to create an Electronic Lab Notebook. Basically, you need to be able to access it via the web, you need to be able to link to images and import tables from Excel and the like. I started doing wiki on a UseMod wiki, so when I started rolling my own, I looked to UseMod, which was abandoned, and then to OddMuse. OddMuse has a few things I like. It's modular, it's Perl, and you don't have to use a whole host of odd modules and root access to make it work.

So, when it came time to make an Electronic Lab Notebook, I thought wiki. I thought OddMuse Wiki.

There's an add-on for tables. There's an add-on for simple image linking.

But....

The suggested WSYWIG editor is FCKeditor. And "fck!" is pretty much how I'm feeling about said editor. Because it isn't writing acceptable output, and I do not for the life of me know why.

Either that, or it isn't interpreting the saved output once the page is rendered again.

Edited To Add: That was it. The outputted wiki-text was incompatable with the wiki, so by using a module that speaks that kind of output, I got it to work. Yay me.

Specifically, I used the Markup.pl extension, in case that helps someone.


Edited To Add: Using Markup.pl kills tables.

Exclaiming "Mea Culpa! Mea Maxima Culpa!" ...

On Thursday, I said I would do one chapter a week.

On Friday, I said I'd do Chapter Two over the weekend.

On Monday, I failed to do anything.

I am in a state of sucking wildly. Suckety Suck Suck Suck.

At least I admit it.

And I'm putting the book on my desk right now.

But I'll mention two things.

First, there's a Perl program that replaces the "READY" text on your office's big HP printer. It is easily adaptable to, for example, take a string of weather information pulled from Weather Underground or whatever source you choose. The author chose Geo::METAR , but I found that too verbose for my purposes. And I am considering creating a module for HP Printer Job Language communication. It'd be my entry into CPAN!

Second, A post on O'ReillyNet brings up an interesting question relating to database inquiries and, as it turns out, a weakness in Perl's sleep. Well, limitation, at least.