Telling One Liners

I don't do a lot of Perl one-liners. In general, I think if you're writing Perl, you're going to want to repeat it so you should put it in a file.

I say in general, because I ran into a specific situation. For a plot-generating process, we 1) pointed to the version of the plot-maker that didn't use libcairo, so it couldn't be used without X11, and 2) accidentally removed lots from the completed file, where we list the plots we've done so we don't redo them, which lead to 3) remaking lots of plots that exist with a plot-maker that couldn't make plots, meaning files that existed got replaced with zero-sized files.

Once we figured out what the problem was, I wrote this one-liner, which ran for over an hour and 40 minutes.

for i in {1..100} ; 
do (
    ls -lS */nanodrop/*png | perl -lane ' print $F[8] if 0 == $F[4] ' | wc -l 
    ); 
sleep 60 ; 
done 

Basically, it looped over and over again to keep constant watch (the for loop), each time taking an ls -l (the -S, order by size, not strictly being necessary but left over from my initial checking), breaking it apart with Perl, printing the file name ($F[8]) if the file size ($F[4]) was 0. I pipe the output into wc -l,  which counts the number of lines. I then sleep for a minute (as the number of zero-length files was over 7000 at first run, the ls-l took quite some time, so the reports were much longer than a minute apart), and do it again.

Granted, I could've done it all in Perl, but why re-implement tools that exist in the shell when you don't have to? I could've done the Perl stuff in Awk, if only I knew Awk better, but I could get to the point where I knew the damage and the rate of repair much faster with tools I knew. Really, the only thing I needed to learn was that perl -lane was what I needed to get the lines broken easily.

Making a CarPuter to step into the New Car

It's been a while since I've written on car computing — over two years, it seems — but that doesn't mean I've stopped thinking about it. The top reasons I would've wanted one in years past would've been for Entertainment and Communication, and I think everyone would agree that today, it's far better to just carry a smartphone than to embed one into your car. And, seeing how smartphones are changing so much so fast, doing more than adding the ability to interface with Bluetooth and/or USB to your vehicle seems silly, at least in short-term.

Navigation doesn't fare too much better, so although it's a toss-up whether dedicated units like Garmin and TomTom are better than smartphone navigation like Google Maps, both are accepted as better (more current maps, better interfaces) than in-dash choices.



I think the use that is least considered is recording. Dash cams are common in Russia because they're used for legal protection, but have given YouTube a great catalog of amazing video. I think there's reason beyond "Hold my beer and watch this" for Americans to have dash cams, and I do want one.

But, ultimately, I think the best reason to get into "Carputers" is Diagnostics, getting into the data that is available from your car's OBDII port. The obvious way is to use an OBDII-to-Bluetooth adapter like the ELM327 or Garmin EcoRoute, but it strikes me that there are enough security vectors in to the New Car that adding more is not a wise route. So, I'm thinking that the Raspberry Pi and an OBDII USB cable might be the better way to handle it, except I'm not sure how to export the data, and while it would be useful to keep track while driving, ultimately, off-the-road analysis is where the usefulness of the process comes in.

I'm thinking that a Raspberry Pi, a cable, a small monitor with composite video and maybe a few other things could be easily turned into a car-monitoring system, and I could pretty easily set something up to only sync with my home network when it's close. I'm not sure whether that's more cost-effective than just getting an ELM327 and an ELM327 app from the Play Store, but I think I'd end up learning more that way.

Anyway, I'm still undecided on the phone/Pi issue, but I think this is something I need to do.

I've got a standing desk, and boy are my legs tired!

Borscht-belt comedy aside, I converted my desk to a standing desk configuration a week ago last Friday, making this the seventh workday where I've spent most of my day standing.

Yes, compared to other standing desks, this is pretty janky. It works for me, though. The tops of my monitors are only a inch or so above eye level, which is pretty close to Hoyle according to ergonomic standards I've seen, and the box put the keyboard at a good typing level. I'm happy with it.

My feet, on the other hand...

My left foot is a bit pronated, which is to say that the ankle turns in a bit and the big toe turns out a bit and the arch is a bit flat.

My right foot? Take every a bit from the previous sentence and put in extremely, and it also lacks the range of motion of my right. I first figured out the issues with my ankles when I noticed I could make fart noises with my foot on the bathroom tile, without trying, but the time I took a hike with my son's Cub Scout pack was when I realized I had a problem, not just an issue.

Since then, I have become much more active. Last year, I did Couch-to-5K, and actually did a 5K. I did it in 45 minutes, finding that I do a mile in 15 minutes or so whether I run or walk, but hey, I got the t-shirt. This year, I've been focused on other activities, but plan to do it again. When I started running last year, I got the same foot pain, which eventually subsided the more I worked. Internet lookup makes me think it's the Extensor Digitorum Brevis muscle that was giving me problems last year, and this last week, this is the pain I've been having again, except this time, in both feet.

The previous pain tells me that I'm doing the right thing, that I'm strengthening my feet and ankles by standing. My recent experience with Cross-Fit tells me that I had better start exercising and strengthening my feet and ankles or this pain won't go away soon.

In terms of productivity, I don't know that I've noticed a difference. I'm in a position at the moment where there's a lot of sit-and-think (or rather, stand-and-think) and not so much great code generation. I don't currently quantify my work, but would be interested in finding a way to do so. Of course, it would've been good to have a body of sitting-desk work in the dataset to use as a comparison....

Stupid Things with References

It took me a while to figure out why I was writing junk to the DB. More or less, this is my code.

my $request = get_request( $request_id ) ;
my $to_wiki = $request ;

I thought to be making a copy of the hash in the hashref.

I meant to be making a copy of the hash in the hashref.

I wasn't making a copy of the hash in the hashref. I was making a copy of the hashref address. Which means every change I made to $to_wiki, I made to $request.

Instead, I needed to do something more like this.

my $request = get_request( $request_id ) ;
my $to_wiki ;
%$to_wiki = map { $_ => $request->{ $_ } } keys %$request ;

I present this as a cautionary tale. Please, learn from the mistakes of others.

Ghosts of Coding Sessions Past

Boss wants to be able to clone a request, to take an item on our list and reset it.

Fair enough, but ...

Requests are among the first things I coded at the lab. Before that, I was at the clinic, and while I did a fair amount of coding there, I was much more of an admin than a developer there, and I have learned a lot about developing here.

Thing is, I learned most of it after I wrote he request code.

A request has lots of data about it. A name, an ID number, who requested it and how to get in contact, what they want done with it, etc. Each request has a number of samples, and the samples have names and numbers and amounts and such. Each sample can have a barcode, which I won't explain. Each request can also have 0-whatever statuses. I back-end to MySQL, and I can copy and massage data, and say in the status that request j is a clone of request i. All that is cool.

Then there's the Wiki.

About everything on the web page is also on the Wiki. It serves as the lab notebook, and I create the stubs. Yes, there's synchronization problems between the Wiki and the database. I try not to think about it, because I cannot find a way for the structured data to interact with the unstructured in a good way.

So, I dread the Wiki stub creation code. I coded it in Perl, using the CGI module. Before I came to the lab, I hand-coded my HTML in my Perl, like my $val = "<div> $foo </div>" which isn't good either, but 200 lines of code like $val = $cgi->div( $foo ) isn't doing my mind any good.

Thrown into the mix, we're moving from our old server to virtual server Sidam, which is Midas spelled backwards, as everything it touches turns to sh*t. I should be moving my code to that server, but that looks to be a bigger issue. To make sure that things finish, we want to move the code as close to as-is as possible. Because it's broken by design (see above) I want to use this opportunity to make great, sweeping changes.

I think I'll implement this with Template Toolkit and suck up the technical debt, but I don't look forward to that process.

Module to automate API form

One thing I've noticed about the web APIs I'm seeing online is that you can often determine what format you want it in. For example, the API URL is something like example.com/api/v1/feed which gives you RSS, but if you want it in ATOM, you can get it at example.com/api/v1/feed/atom. Want JSON? example.com/api/v1/feed/json. This way, your Javascript can AJAX that mess up, while your users can pull it into EXCEL with example.com/api/v1/feed/csv and be happy.

This is cool. So I wrote some Perl to automate that stuff for me.

This does not do JSONP, because I'm not expecting, right now, to want to cross-site script the stuff I'm exporting right now. But maybe. And it shouldn't be too bad of a modification. And this is a naive and potentially stupid way of doing CSV. Maybe wrapping everything in quotes?

The cool step would be to make a dispatch table and function refs to the right thing, pulling the magic for each format out. This might be done later.

MySQL -> R

Here is how you pull SQL data into R.

Cool, isn't it?

Except there's a password in plain text. Which means I have to modify it when I want to show it off.

I'll have to work on that.

This is how I'm stupid

I wrote a tool about four months ago, which is long enough that I barely remembered writing it, much less what I had done in it. It connects the data coming out of one of our instruments with the metadata about the samples we fed into it. Part 1 of this is to create a comment field which is easily pushed onto the instrument's data output. The newest part takes the data images from this and combines them with the sample metadata to be published into our electronic lab notebook.

One of the cool bits about this is that, if you are creating plate records that can take 11 samples, but you have 38 samples, it will create four plate records; three with all 11 samples filled and one with just five samples.  

This is where we get into problems. Those four plate records are logically joined, with the thought that you putting them together means they go together. So, the samples are numbered between 1 and 38, in this case. Problem is, samples coming off the instrument are sample 1 through sample 11. So, sample 12 and on just fall on the floor.

I am also stupid because I put the fact that this plate can hold 11 samples into the comment field block but not independently into the database table. 

For four months, nobody who grouped several plates together like this used the tool to publish them to the lab notebook, so nobody found how I was stupid. 

Yes, I am saying I was stupid. Stupid is as stupid does, as Forrest Gump's mama said, and this is two stupid things I've done. I'll throw in "You didn't test it enough" as a third.

It isn't helpful to just say "You're stupid". It also isn't useful to say "You're stupid for using (Tool A which I don't like) instead of (Tool B which I do like)" because people do stupid things in the best of languages and do smart things in the worst. I believe it is helpful to identify the places where you are stupid, like I do above, because those are actionable. I can now add a column to the table to store the number of samples on a plate, which I might do. Right now, I'm using a regular expression to pull the number of samples out of the comment, then using modulus to find the adjusted sample number, and then saying sample number = number of samples if sample number == zero. Yeah, I'll have to add a "sample number on plate" column into the sample database table, too.

I'm glad to know these things. I'm glad they're now patched, and I plan to fix these things better. Mostly, I'm glad to know where I'm stupid because now I know how to be smart next time.

Unboxing Day - Canon EOS Rebel T3

Our old Gel Documentation System runs Windows 98, and the hard drive sounds like an abused Yugo. So, we're replacing it. Instead of going with custom hardware and software, we're running with a COTS DSLR. Today I got the camera. Soon I should get the charger. Once I prove that I can take the shots I want in Windows 7 with the Canon software, I'll order the correct filter and we'll rock the new stuff. 

Thinking Through An Idea - Caffeine Tracker

I tend to get horrible headaches on Saturday. I believe it is in part because I've started to drink coffee only on weekdays, only on workdays and only during work hours (9am-5pm) and Saturday is when the caffeine withdrawal hits. At least, that's my theory.

So, I'm considering making an Android app where I store each caffeinated beverage I drink. At first pass, I can get the number of cups and when. Second pass, I should be able to extrapolate caffeine amounts and start to figure out when it's in my system and when it isn't. I can work out when I tend to drink lots and when I tend to drink little.

I could write this as an HTML5 thing. There's certainly a case to be made for that, but I wouldn't be learning a vastly different skill set on that one.

What I want to do is make an Android app. It would be vaguely cooler to write the Java itself, but the first step might be to write it in HTML5, do it in PhoneGap, then push it to Android. Moving to iOS and Windows Phone and others (OpenWebOS?) should be easy. But then, I'm not learning Java. I'm not adding another skill to my resume.

So, this'll be an Android native app. More as I get to it.