That Was Dumb, or the Dangers of Not Packratting

So, there's this instrument. We have a bunch of samples going into the instrument, and we pull data out which tells us whether it is worth it to run the samples on the next, more expensive instrument. So, it's for quality assurance.

I do several things with this thing. I have an interface that collects all the information about group of samples (what goes in which lane, etc.) which we import after the sample is run to connect name, ID numbers and all sorts of metadata with each sample. Right now, there's a comment section, and I put together a CSV table of the metadata in there. The database table I created is set to be all the data fields that the instrument software takes, which means I create the comment on Submit. And it means I don't save all the metadata independently.

This, I am finding out, is stupid.

Because I also code stuff handling the output. Before, I simply used Sikuli to automate the export of the XML, but now I'm handling exported graphs and putting them into our lab notebook, and I'm looking at having to deconstruct the comments in order to get the information I need.

Knuth was right. Premature optimization is the root of all evil.

Oh, That Was Clever! (WebDev)

I write this in Blogger, and there's a helpful icon I can press to upload an image.

I run a wiki for the lab. That wiki does not have a helpful "upload an image" button. I've long seen that as a failing.

Turns out, my users found a work-around.

We use CKEditor as our interface because it would be cruel to force others to understand wiki markup, and because you can copy rows and columns from Excel and it turns them into tables.

Guess what else you can do?

You can create things in Word, copy them, and paste them into CKEditor, which saves them as data URLs.

On the good side, you can get your images onto the wiki.

On the bad side, it blows the top off the page size limits, which are put there for your protection.

I'm seriously going to have to find a solution to this, because this is seriously causing problems. But, honestly, once I got what was going on, I laughed out loud.

Making the RCA Media Player Usable

ElectronicHouse.com

This is an RCA Wi-Fi Streaming Media Player. I got it when I got a Sony HDTV for the bedroom.

I have two comments on that purchase:

The remote for this RCA player is utter crap. And two HDMI ports will not be enough.

Tekzilla has pointed me to the solution to the second problem. You can now get HDMI switches that auto-switch, which means, for example, you have the cable box on port 1, the Blu-Ray player on 2 and the RCA Faux-ku on 3. You generally keep the DVR-having cable box on and the other two off, so, when you turn on the Blu-Ray, it auto-switches to that, and switches back when done. I need one. But that's not the point today.

The point today is that the remote for the RCA Streaming Media Player is crap. And the button battery it comes with is weak. But, thankfully for me, I can program my Logitech Harmony 200 remote to talk to it. And, while it is not the best of the Harmony line — I got the low-powered one as a test and preview for the bedroom before I got a better one for the living room — it is quite a lot better than the one that came with.

Honestly, I wish I had gone with a Roku, but I didn't. I'm reasonably happy with the unit (Pandora and Netflix work fine, Hulu less so). But the first issue with this device is now rectified.

Big Three for Sept 10-15

• TEDx - I have an idea for replacing the scroller images (which make Purdue look like a concentration camp) with images (and links to the associated videos) from last year's event. Also, seeing if we can do PHP on the server.
• FFH, or Just For Me - develop a small web app using something like Google Places and the HTML5 Location API to tell me what restaurants are close. 
• Self - When I found that my tendons are weird, leading to flat feet and bunions and foot pain, I stopped running (except for a 5K, which I mostly walked). I half-heartedly decided that I would do five push-ups and five sit-ups every day. The push-ups stopped after the first day -- low upper-body strength meets 240+ weight -- and after doing MWF last week, I could feel the muscles aching for days after. So, I think I'll go with MWF. Which means I have to remember to do some sit-ups when I get home today. 

Additional to-do: Remember to get this done on Friday. I was in a weird mood for no good reason that day.

Big Three for September 3-7

Here we are, at it again. I'm writing down my Big Three tasks for the coming week. I think I'll admit the progress on the previous Big Three.

• Main Work: This one is actually pretty good. I've had some feature requests and I'll get some more, but I have it in good condition. 
• TEDx Web Development: Institutionally, we're starting to get together, which is good for the team. We're starting to recruit people to take on pieces. This is good. And I have tried to put a basic hello_world.cgi onto the Dev server, and it failed. This is ... well, at least I know. There's a code review process, so I have to start looking into the "who do I ask to get this going" problem to try to get server-side code on this project.
• Food For The Heart: The group has made progress. There is a meeting with a possible funding source. There was a great meeting a week ago where our "Angels" told us what they need from us before they'll cut checks. In terms of institutional setup, I'm happy. But in terms of programmer work, we're not at a point where I can start doing anything.

So, that's my progress. So, what's my next set of goals?

• Main Work: I had been planning on taking a PC and turning it into my Linux desktop, then taking my old Linux desktop and turning it into a Git server. On Thursday, I finally hit the point where I had time, and I made a bootable thumbdrive and everything. And today I find that something on my desktop is kicking out spam. So, this is the justification I needed to move forward. And, I need to look up firewalls and cfengine.
• TEDx Web Development: There will of course be the process of finding the team. My #1 goal, though, is to think through the desired look and functionality.
• Food For The Heart: I need to keep part of the business-building process, because that's the part I most need to learn. But, so I can keep my interest up, I'll probably try to make a location-based app of some sort, just to jump-start my skills on the subject.

What do I do with my weather stuff now?

It started with me wanting to know how to code with R. R is about handling and graphing data, and I didn't really have data of my own. So, what I did was find a source I liked for weather and started putting it into a MySQL database.

That source was Google, specifically the XML used for weather in the soon-to-be-closed iGoogle.

Here's that code.

#!/usr/bin/perl

# $Id: hpsetdisp.pl 11 2006-03-22 01:21:03Z yaakov $

# hpsetdisp.pl
# Connects to a JetDirect equipped HP printer and uses
# HP's control language to set the ready message on the
# LCD display.  Takes an IP address and message on the
# command line. My favorite message is "INSERT COIN".
# Keep in mind the limitations of the display when composing
# your clever verbiage.
#
# THIS PROGRAM IS PROVIDED WITH NO WARRANTY OF ANY KIND EXPRESSED OR IMPLIED
# THE AUTHOR CANNOT BE RESPONSIBLE FOR THE EFFECTS OF THIS PROGRAM
# IF YOU ARE UNCERTAIN ABOUT THE ADVISABILITY OF USING IT, DO NOT!
#
# Yaakov (http://kovaya.com/ )
#
# Modified by Garrett Hoofman
# 10/18/2007
# http://www.visionsofafar.com

# Modified by David Jacoby
# 12/10/2008
# http://varlogrant.blogspot.com/

# Modified by David Jacoby, using notify-send and tweaking Unicode
# 09/08/2008
# http://varlogrant.blogspot.com/

# Modified by David Jacoby, moving from Google to NOAA
# 08/28/2012
# http://varlogrant.blogspot.com/


use strict ;
use warnings ;
use Modern::Perl ;
use DBI ;
use Getopt::Long ;
use IO::Interactive qw{ interactive } ;
use IO::Socket ;
use XML::DOM ;
use Data::Dumper ;

use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;

use subs qw{  get_weather notify db_upload } ;

my $zip         = '' ;
my $interactive = '' ;
my $degree = '°' ; #the unicode, straight
   $degree = "\x{00b0}" ; #the unicode, specified
my $rdymsg      = "Ready" ;
my $curr_cond ;

GetOptions(
    'zipcode=s'   => \$zip ,
    ) ;

$zip !~ /\d/ and exit 0 ;
my %weather = get_weather ;
db_upload %weather ;

exit ;

########## ########## ########## ########## ########## ########## ##########
sub get_weather {
    my %output ;
    my $parser = XML::DOM::Parser->new() ;
    my $file   = 'http://www.google.com/ig/api?weather=XXXXX&hl=en' ;
    #my $zip    = shift ;
    $zip =~ s{(\d{5})}{}mx ;
    $zip = $1 ;
    $file =~ s/XXXXX/$zip/mx ;
    my $doc = $parser->parsefile( $file ) ;
    exit if ! defined $doc ;
    $output{zip}        = $zip ;
    $output{city}       = $doc->getElementsByTagName( 'city' )->item( 0 )
                        ->getAttribute('data');
    $output{temp_c}     = $doc->getElementsByTagName( 'temp_c' )->item( 0 )
                        ->getAttribute('data');
    $output{temp_f}     = $doc->getElementsByTagName( 'temp_f' )->item( 0 )
                        ->getAttribute('data');
    $output{humidity}   = $doc->getElementsByTagName( 'humidity' )->item( 0 )
                        ->getAttribute('data');
    $output{wind}       = $doc->getElementsByTagName( 'wind_condition' )->item( 0 )
                        ->getAttribute('data');
    $output{conditions} = $doc->getElementsByTagName( 'condition' )->item( 0 )
                        ->getAttribute('data') ;

    return %output ;
    }
########## ########## ########## ########## ########## ########## ##########

########## ########## ########## ########## ########## ########## ##########
sub db_upload {
    my %cond = @_ ;
    my $dbh = db_connect() ;
    for my $k ( sort keys %cond ) {
        $cond{$k} = $dbh->quote( $cond{$k} ) ;
        }
    my $fields = join ',' , qw{
        city        conditions
        humidity    temp_c
        temp_f      wind
        zip
        } ;

    my $sql ;
    my $rows ;

    $sql = qq{
    INSERT INTO weather_data ( $fields ) VALUES (
        $cond{ city } ,
        $cond{ conditions } ,
        $cond{ humidity } ,
        $cond{ temp_c } ,
        $cond{ temp_f } ,
        $cond{ wind } ,
        $cond{ zip }
        ) } ;
    #$rows = $dbh->do( $sql ) or croak $dbh->errstr;

    $sql = qq{
    INSERT INTO weather ( $fields ) VALUES (
        $cond{ city } ,
        $cond{ conditions } ,
        $cond{ humidity } ,
        $cond{ temp_c } ,
        $cond{ temp_f } ,
        $cond{ wind } ,
        $cond{ zip }
        ) } ;
    $rows = $dbh->do( $sql ) or croak $dbh->errstr;

    return ;
    }
########## ########## ########## ########## ########## ########## ##########

Problem is, Google knows I've been grabbing weather data every 10 minutes and has blocked me.

While I fill in all these, I am not pulling in weather data from multiple cities, so I can hardcode city and zip, and I couldjust drop conditions and wind. All I'm really watching is temp_f, and it's a simple conversion to get temp_c if I wanted it.

I have code that pulls from NOAA. Think I'll try to bring that in here.

Weight Graphs

These are a series of generated graphs. The jagged blue line is the raw data of my more-or-less daily weight measures, and the red line is an average of the previous days' measurements. 

I don't measure every day, because I am human and therefore fallible. I take an average of the measurements of the n previous days, which means if I'm trying to get 20 and I skipped twice, I get 18. I would like to have that number graphed here, but I don't know R that well yet. Here, I'm increasing n by five, going from 5 to 30. My goal is to get to the trend line, to show I'm losing weight in total without getting caught up in the day-to-day jagged differences. After some poking, I decided to go with 20 days. Here, I've run the graph several times with different levels so I can see if there's anything better.

Clearly, the first few weeks will look similar, because a five day average will look like a 30 day average if you only have five days to work with. The trend line approximates 1 lb a week, I think. I should add that as a third line.

Big Three for August 27-31

1. TEDx Web Development: I do web for TEDxPurdueU and so far, that has meant maintenance work. I need to get in front of it, come up with an alternate look, and go from there, so we can toss the old and embrace the new. So, specifically, I want to have a front page that demonstrates my desired look. Plus, clear out backlog, and know to what extent we can run dynamic pages on the server.
2. Food For The Heart: This is the "Do that work" mentioned in the previous. I have taken the visual plan from whiteboard to static HTML prototype. I need to look up and use an icon set, sprite or font, set it up for Bootstrap, and add some features.
3. Main Work: I am convinced that my API design is stupid and needs redesign. I have moved the application code for APR into libraries, so I can change config in the main page and easily move between test and production. I need to move all the output from several pages to one, where the caller object says "operator_list" to the API core to get data rather than looking to ajax_operator_list.cgi

If you see me, feel free to ask me how I'm doing on my Big Three.

My Next Personal Project

I don't like distractions. They're distracting.

I also don't like missing a mail from my boss. They're rare, so when they come, they're often important.

So, I shut down alerts in Thunderbird and wrote a tool called jBiff.

jBiff connects to your mail server via IMAP and finds the new mail (thus biff) that meets the standards you want to watch for, then sends an alert via XMPP, aka Jabber (thus j). As is, I'm mostly searching on sender, and I'm not sure, but that might be all that works.

I suppose I should put this on Github. I'm pretty sure that step's overdue. But there's a problem.

jBiff uses IMAP. There are certain mail servers that don't offer IMAP. This obliges me to start working in POP.

And, there's another possiblity. There are many places I receive mail but never check. There are two primary places I check mail, and if you know me, you know what they are. But having a tool to check the status of the others and notify me weekly or so, that would be good.

Easy Does It!

As previously blogged, my Stapes Easy button has been turned into a switch and connected to a Teensy. The last issue was getting the code worked out. I now have that code worked out.

And released to Github.

// ========= ========= ========= =========
void setup() {
  // make pin 10 an input and turn on the 
  // pullup resistor so it goes high unless
  // connected to ground:
  pinMode(10, INPUT_PULLUP);
  delay(2000);
  Keyboard.begin() ;
  }

// ========= ========= ========= =========
void loop() {
  if ( digitalRead( 10 ) == LOW ) {
    Keyboard.press( KEY_SCROLL_LOCK ) ;
    Keyboard.releaseAll() ;
    delay(300); 
    Keyboard.press( KEY_SCROLL_LOCK ) ;
    Keyboard.releaseAll() ;
    delay(600); // Delay of 6/10 second to keep from repeating.
    } 
  delay(10);
  }

This was my first serious use of Github for Windows, too. Far easier to handle than the previous stuff I did.