Stupid Things with References

It took me a while to figure out why I was writing junk to the DB. More or less, this is my code.

my $request = get_request( $request_id ) ;
my $to_wiki = $request ;

I thought to be making a copy of the hash in the hashref.

I meant to be making a copy of the hash in the hashref.

I wasn't making a copy of the hash in the hashref. I was making a copy of the hashref address. Which means every change I made to $to_wiki, I made to $request.

Instead, I needed to do something more like this.

my $request = get_request( $request_id ) ;
my $to_wiki ;
%$to_wiki = map { $_ => $request->{ $_ } } keys %$request ;

I present this as a cautionary tale. Please, learn from the mistakes of others.

Ghosts of Coding Sessions Past

Boss wants to be able to clone a request, to take an item on our list and reset it.

Fair enough, but ...

Requests are among the first things I coded at the lab. Before that, I was at the clinic, and while I did a fair amount of coding there, I was much more of an admin than a developer there, and I have learned a lot about developing here.

Thing is, I learned most of it after I wrote he request code.

A request has lots of data about it. A name, an ID number, who requested it and how to get in contact, what they want done with it, etc. Each request has a number of samples, and the samples have names and numbers and amounts and such. Each sample can have a barcode, which I won't explain. Each request can also have 0-whatever statuses. I back-end to MySQL, and I can copy and massage data, and say in the status that request j is a clone of request i. All that is cool.

Then there's the Wiki.

About everything on the web page is also on the Wiki. It serves as the lab notebook, and I create the stubs. Yes, there's synchronization problems between the Wiki and the database. I try not to think about it, because I cannot find a way for the structured data to interact with the unstructured in a good way.

So, I dread the Wiki stub creation code. I coded it in Perl, using the CGI module. Before I came to the lab, I hand-coded my HTML in my Perl, like my $val = "<div> $foo </div>" which isn't good either, but 200 lines of code like $val = $cgi->div( $foo ) isn't doing my mind any good.

Thrown into the mix, we're moving from our old server to virtual server Sidam, which is Midas spelled backwards, as everything it touches turns to sh*t. I should be moving my code to that server, but that looks to be a bigger issue. To make sure that things finish, we want to move the code as close to as-is as possible. Because it's broken by design (see above) I want to use this opportunity to make great, sweeping changes.

I think I'll implement this with Template Toolkit and suck up the technical debt, but I don't look forward to that process.

Module to automate API form

One thing I've noticed about the web APIs I'm seeing online is that you can often determine what format you want it in. For example, the API URL is something like example.com/api/v1/feed which gives you RSS, but if you want it in ATOM, you can get it at example.com/api/v1/feed/atom. Want JSON? example.com/api/v1/feed/json. This way, your Javascript can AJAX that mess up, while your users can pull it into EXCEL with example.com/api/v1/feed/csv and be happy.

This is cool. So I wrote some Perl to automate that stuff for me.

This does not do JSONP, because I'm not expecting, right now, to want to cross-site script the stuff I'm exporting right now. But maybe. And it shouldn't be too bad of a modification. And this is a naive and potentially stupid way of doing CSV. Maybe wrapping everything in quotes?

The cool step would be to make a dispatch table and function refs to the right thing, pulling the magic for each format out. This might be done later.

MySQL -> R

Here is how you pull SQL data into R.

Cool, isn't it?

Except there's a password in plain text. Which means I have to modify it when I want to show it off.

I'll have to work on that.

This is how I'm stupid

I wrote a tool about four months ago, which is long enough that I barely remembered writing it, much less what I had done in it. It connects the data coming out of one of our instruments with the metadata about the samples we fed into it. Part 1 of this is to create a comment field which is easily pushed onto the instrument's data output. The newest part takes the data images from this and combines them with the sample metadata to be published into our electronic lab notebook.

One of the cool bits about this is that, if you are creating plate records that can take 11 samples, but you have 38 samples, it will create four plate records; three with all 11 samples filled and one with just five samples.  

This is where we get into problems. Those four plate records are logically joined, with the thought that you putting them together means they go together. So, the samples are numbered between 1 and 38, in this case. Problem is, samples coming off the instrument are sample 1 through sample 11. So, sample 12 and on just fall on the floor.

I am also stupid because I put the fact that this plate can hold 11 samples into the comment field block but not independently into the database table. 

For four months, nobody who grouped several plates together like this used the tool to publish them to the lab notebook, so nobody found how I was stupid. 

Yes, I am saying I was stupid. Stupid is as stupid does, as Forrest Gump's mama said, and this is two stupid things I've done. I'll throw in "You didn't test it enough" as a third.

It isn't helpful to just say "You're stupid". It also isn't useful to say "You're stupid for using (Tool A which I don't like) instead of (Tool B which I do like)" because people do stupid things in the best of languages and do smart things in the worst. I believe it is helpful to identify the places where you are stupid, like I do above, because those are actionable. I can now add a column to the table to store the number of samples on a plate, which I might do. Right now, I'm using a regular expression to pull the number of samples out of the comment, then using modulus to find the adjusted sample number, and then saying sample number = number of samples if sample number == zero. Yeah, I'll have to add a "sample number on plate" column into the sample database table, too.

I'm glad to know these things. I'm glad they're now patched, and I plan to fix these things better. Mostly, I'm glad to know where I'm stupid because now I know how to be smart next time.

Unboxing Day - Canon EOS Rebel T3

Our old Gel Documentation System runs Windows 98, and the hard drive sounds like an abused Yugo. So, we're replacing it. Instead of going with custom hardware and software, we're running with a COTS DSLR. Today I got the camera. Soon I should get the charger. Once I prove that I can take the shots I want in Windows 7 with the Canon software, I'll order the correct filter and we'll rock the new stuff. 

Thinking Through An Idea - Caffeine Tracker

I tend to get horrible headaches on Saturday. I believe it is in part because I've started to drink coffee only on weekdays, only on workdays and only during work hours (9am-5pm) and Saturday is when the caffeine withdrawal hits. At least, that's my theory.

So, I'm considering making an Android app where I store each caffeinated beverage I drink. At first pass, I can get the number of cups and when. Second pass, I should be able to extrapolate caffeine amounts and start to figure out when it's in my system and when it isn't. I can work out when I tend to drink lots and when I tend to drink little.

I could write this as an HTML5 thing. There's certainly a case to be made for that, but I wouldn't be learning a vastly different skill set on that one.

What I want to do is make an Android app. It would be vaguely cooler to write the Java itself, but the first step might be to write it in HTML5, do it in PhoneGap, then push it to Android. Moving to iOS and Windows Phone and others (OpenWebOS?) should be easy. But then, I'm not learning Java. I'm not adding another skill to my resume.

So, this'll be an Android native app. More as I get to it. 

Pinokio, or, Oh my goodness I have to make this

Pinokio from Pinokio on Vimeo.

My friend Mark sent me this, and he was right. It's right up my alley. It is the next step from webcam-on-a-swing-arm idea. I really need to start learning how to add servomotors to a lamp and start making this myself.

I love how it's also a call-back to the World's Most Useless Device, except, instead of always turning itself off, it turns itself back on.

LED Light Bars

I love this idea. I drive a Toyota Yaris, which isn't cool enough for Eagle Eye LED Light mods.

So, I need to do something with the idea. I don't know what I'd want to do with it, but I want to.

You know, I might just start with a straight one, make it a light stick, and work on the use at a later time.

Randy did suggest that this might be useful toward the sunrise lamp idea, and there is something to that.

Radical Change in Higher Education - CS for Everyone!

I was pointed to a blog post titled Radical Change in Higher Education: Money, Sports and Computers. The author presents a distrust in Massive Open Online Courses (MOOCs) like Udacity, then goes on to present her ideas for deeper, more radical change.

There are things I kinda agree with but don't forsee happening, such as the ending of college sports in the way they exist now and the creation of minor leagues in their stead. I never really thought it made sense to associate institutes built upon intellect to associate so tightly with teams built on physical prowess. The minor leagues of baseball exist because the baseball is a summer sport and college students are generally off over the summer, so collegiate sports cannot attract the same audiences. Summer break exists because summer is when people were needed back at home, and the low number of people involved in agriculture makes that no longer a design requirement, so the fundamental change I would make, trimesters, would tend to make collegiate baseball more viable and minor league baseball less viable. Ah well.

Unless I misunderstood something, I believe that my school's sports program is self-supporting. I think that, if a university's program is not self-supporting, or if there isn't something else that the school gets from it that justifies it (and for the life of me, I can't think of anything, but I've never been a team sports guy), it would be better if it got rid of it.

I see the point of "replacing" collegiate sports with fitness and wellness programs, but honestly, every personal improvement in wellness I have ever experienced has come from working with myself, not with a group. I think the group dynamic messes it up, but that might just be me.

The last point, the one I'll quote, is one I strongly agree with and disagree with.

Computer Science: CS should be required. For everyone.  Can you be a historian today without using a computer?  An artist?  A salesperson?  Anything?  Shouldn’t we aspire to turn out a new generation of educated men and women who have more than a surface knowledge of how the blasted things work, since their success in no small part will depend on that knowledge?

I hold a CS degree. I work with computers in the day and play with them at night. My day work with computers involves the use of computers in science, and I've been saying this for years: Today, all science is computer science, because the research that could've been done without computers has been done without them already. I think that the same is becoming true in other fields. Between Processing, Tod Machover's work and work with genetic algorithms in composing, there's precedent for the use of computational tools in the arts. I think you can still be a historian without using a computer much more than using it for email and word processing, but I've heard of historians making more interesting use of it. First, there's the wider dissemination of contemporaneous source material, but beyond that, many are beginning to see digitized libraries as a Big Data source, where you can graph the rise and fall of ideas and people by the number of occurrences of them in the text.

I'd throw in the idea that this goes down to the level of skilled labor. Adam Davidson writes in the New York Times magazine about the "Skills Gap", saying that machinist training starts with using tools to cut metal but quickly move on to computer-aided methods.

So, yes, I'm big in agreeing that there's value in a great many fields in embracing the computer revolution. I'm all for teaching programming to all. I'm just not sure that Computer Science is really where you want that.

Computer Science is different. Computer Science, properly considered, is a branch of mathematics built around complexity. Yes, students go into Computer Science as a step toward becoming programmers, but this means there's a great deal of knowledge they gain and never use, and a great deal of knowledge they don't gain until they get on the job and find they need it. I still feel undertrained in the use of version control, for example. Those would be better served with a program built around Software Engineering, and that term is problematic, as there is no mechanism for becoming a licensed software engineer, but are required to call yourself an engineer in other fields, and many of the greatest advances in computing come from people who have the most tenuous claim to that title. Linus Torvalds was a college student when he started developing Linux.

Consider the case of databases. I would consider that there is one set of skills where users (be they artists, historians, scientists, programmers, machinists...) might use to collect what they need, another set of skills that programmers use to effectively create the tables and data structures to be used by others, and another set of skills which programmers use to create the database engines themselves. The first set of skills are things I would wholeheartedly endorse encouraging everyone to know. The second set is a little less useful unless you're stepping up to start collecting your own data. I'm learning it a little at a time, and finding holes in my knowledge every time I create a set of tables. The third set, the skills that those at Oracle or in Microsoft's SQL Server team or involved in PostgreSQL develop, are ... I don't know enough about them to really describe. But it's more about how to make these things fast at the hardware level, so you can have several thousand transactions a second go through when under load.

Thing is, while the last category is closest, none of it is really Computer Science. I think forcing this association between computer use and Computer Science doesn't help any party involved.