Gone Google Docs

Yeah, I've gone Google. This is a spreadsheet that I'm maintaining, showing my fuel efficiency. When I get gas, I take webcam photos of the dash and the pump, then transfer the odometer and gas pump data to this spreadsheet. I started using Google Docs for this because my netbook has a 10GB SSD in it and just XP and the few apps on it take up most of that. If I keep it all on Google Docs, then I don't have to have either Office or OpenOffice (and thus Java) on my dinky netbook.

Columns A-F are straight reporting. G is subtraction, and then miles/gallon and miles/day are simple division. I think I can get more interesting and better data out of this when I learn a cooler set of Excel-like spreadsheet commands.

I kinda wish I could do this from my phone. I can't, though. My smartphone isn't smart enough. Anyone know if GoogleDocs works under the iPhone?

Addendum

I have R 2.9.2 plus Revolution-R on my Linux box.

The sample code works.

But I didn't test it before installing Revolution-R.

And I don't remember if this was a problem under Linux when I first did this three years ago. Which it might not have.

Which makes it a bit of User Fail.

Generating PNGs or JPGs via crontab in R

Dealing with R is one of the requirements of the position I have here in the shower, and here is a long-lasting complaint. We get data that we want to plot. R is wonderful for that. Here's a bit of example code.

temp <- c(63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 65 , 64 , 64 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 62 , 62 , 62 , 62 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 59 , 59 , 59 , 59 , 59 , 59 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 57 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 55 , 55 , 55 , 55 , 55 , 55 , 55 , 55 , 54 , 54 , 54 , 53 , 53 , 53 , 53 , 53 , 53 , 52 , 52 , 52 , 52 , 52 , 52 , 52 , 52 , 52 , 53 , 52 , 52 , 52 , 52 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 )
png("/home/jacoby/test_weather.png")
plot( temp , type="l" )
dev.off()

That's a big, ugly code block, isn't it? That is actually supposed to be a graph of the weather info for the last 24 hours, taken from the Google Weather API in 20-minute intervals, but it mimics what I want graphed. There's some text shoved into the real example, but no big. In the real example, I have the template in a chunk of Perl code, which Perl fills with the correct values and then runs it as a shell command, which looks a bit like this.

R CMD BATCH /home/jacoby/weather.R 

When I run it by command line, that works perfect. I get a pretty PNG. Actually pretty ugly, but I could learn more formatting and pretty it up. But, when I run it via command line, we get this for the weather.Rout.

R version 2.5.0 (2007-04-23)
Copyright (C) 2007 The R Foundation for Statistical Computing
ISBN 3-900051-07-0

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Previously saved workspace restored]

> temp <- c(63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 66 , 65 , 64 , 64 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 63 , 62 , 62 , 62 , 62 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 61 , 59 , 59 , 59 , 59 , 59 , 59 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 58 , 57 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 56 , 55 , 55 , 55 , 55 , 55 , 55 , 55 , 55 , 54 , 54 , 54 , 53 , 53 , 53 , 53 , 53 , 53 , 52 , 52 , 52 , 52 , 52 , 52 , 52 , 52 , 52 , 53 , 52 , 52 , 52 , 52 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 , 51 )
> png("/home/jacoby/test_weather.png")
Error in X11(paste("png::", filename, sep = ""), width, height, pointsize,  : 
 unable to start device PNG
In addition: Warning message:
unable to open connection to X11 display '' 
Execution halted

Notice how it's calling X11? If I run it via command line, it has access to X. If I run it via crontab, it doesn't. Which makes me wonder what sort of idiot decided to use that library for PNG generation. But that's not my core wonder. My core wonder is, is there a workaround? I mean, one better than our current "generate Postscript and use ps2jpeg to convert? Because for text and lines, for graphs, really, PNG is better than JPEG, and generating the right thing is better than several layers of conversion.

Yeah, it's a fairly ancient version of R. I think current is 2.11. It runs on a big Sun machine shared by several people whose grants pay for it and such, and if it isn't broken, think twice about an upgrade, right? I'd be hesitant to beg for an upgrade unless I was positive this was the fix.

Shub-Internet, tell me, is there an easy way to generate PNGs in R without having to use the X11 library?

Refactor Question relating to an ancient protocol for transferring files

I work in a lab, and we have lots of instruments that do highly-complex technical things. Sometimes they spit out gobs and gobs of data, and sometimes they don't. We have lots of computers connected to those instrument machines. We call them instrument machines, and when we have an instrument that spits out lots of data, we have an FTP server that allows the main server to come it at scheduled times and check for new files.

We had a program that would go and get all the data off said machine in a serial manner. It looked pretty much like this:

my $ftp = Net::FTP->new(login stuff) ;
$ftp->binary() ;
my @list = $ftp->ls() ;
my @files ;

for ( @list ) {
    if( ! $downloaded{$_} ) {
        $ftp->get($_) ;
        push @files , $_ ;
        }
    }

for my $file ( @files ) {
   open my $fh , '<' , $file ;
   while (<$fh>) {
       chomp ;
       my @line = split m{\t} , $_ ;
       dump_into_database( $line[0] , $line[3] , $line[27] , $line[491] ) ;
       }
   $downloaded{ $file }++ ;
   }

Yes, it has been substantially simplified.

There are two problems with this program. The original, not the simplified version shown here. First, there's no clear separation between sections, which makes figuring out what each specific part is doing awful confusing. Second, it was modified and hacked upon several times by a clueless idiot.

(waves hands.)

My thought is to have everything in subroutines, each small enough to fit in my head and screen-height window. And I am doing it now.

sub handle_ftp {
    my $ftp = Net::FTP->new(login stuff) ;
    $ftp->binary() ;
    my @list = $ftp->ls() ;
    my @files ;
    for ( @list ) {
        download( $ftp , $_ ) ;
        }
    }

sub download {
    my ( $ftp , $file ) = @_ ;
    if ( !$downloaded{ $file } {
        $ftp->get( $file ) ;
        split( $file ) ;
        $downloaded{ $file } ++ ;
        }
    }

sub split {
    my ( $ftp , $file ) = @_ ;
    open my $fh , '<' , $file ;
    while (<$fh>) {
        chomp ;
        my @line = split m{\t} , $_ ;
        dump_into_database( $line[0] , $line[3] , $line[27] , $line[491] ) ;
        }
   }

My concern is that, previously, we're going through the FTP session as fast as possible, while in the second case, the FTP session is going until the last split is done. Here, where my car is parked further away from the instrument machine than the server room is, and it's all reasonable small text files, so run time shouldn't bee too complex. There' actually three variations on dump_into_database, including one that generates an image in R.

I think I like the second better because it seems more recursive, even through it really isn't.

Without dumping the whole thing here, does anyone see a problem with my approach?

I Need More Mojo

We have an instrument machine. What that means is, we have a dingus that's computer-controlled, and we have a computer to control the dingus. This dingus is complex, shiny and new, and this machine has Windows 7 64-bit and 16 gigs of RAM. Easily the swankest machine in the lab, but since we don't use the instrument nearly at all, we don't use the machine at all.

We had occasion today to use it today. Specifically, there was CSV that someone wanted to open in the use-within-30-days initial Office Excel 2007. And it's been well past 30 days. We have a site license, so I grab the discs.

It will not recognize the 2007 disc as a disc, even, but 2003 goes on easily. So I install Office 2003, check for updates, let it reboot, and try again.

No love for the Office 2007 install media.

I have a netbook w/o CD drive so I put an external USB CD drive on my Christmas list. Shazam! It works.

So, I get my boss. "Hey, boss. This is flaky. See how it works here?" I take the CD out of my drive and into the machine's internal drive.

And it works perfectly

I swear, this is the second time this has happened to me this month! He just stands near misbehaving hardware and it sees the error of it's ways. I need this aura of fix. I need more freakin' mojo.

Addendum To Someone Else's Work

Seven JavaScript Things I Wish I Knew Much Earlier In My Career is the work. I've been poking at it for a little while, reading it while I'm not working hard on another problem.

From this HTML. you want to call functions rather than go to links.

Great Web resources
  
Opera Web Standards Curriculum
  
Sitepoint
  
A List Apart
  
YUI Blog
  
Blame it on the voices
  
Oddly specific

Author's first example code:

// Classic event handling example
(function(){
  var resources = document.getElementById('resources');
  var links = resources.getElementsByTagName('a');
  var all = links.length;
  for(var i=0;i<all;i++){
    // Attach a listener to each link
    links[i].addEventListener('click',handler,false);
  };
  function handler(e){
    var x = e.target; // Get the link that was clicked
    alert(x);
    e.preventDefault();
  };
})();

Author's second example code:

(function(){
  var resources = document.getElementById('resources');
  resources.addEventListener('click',handler,false);
  function handler(e){
    var x = e.target; // get the link that was clicked
    if(x.nodeName.toLowerCase() === 'a'){
      alert('Event delegation:' + x);
      e.preventDefault();
    }
  };
})();

My take, assuming jQuery:

$( function() {
    $( '#resources li a' ).click( function() {
        alert( 'Event delegation: ' + $(this).attr('href') ) ;
        return false ;
        } ) ;
    } ) ;

Unless I miss a trick, it works the same and uses far fewer lines.

So, 1000 words each?

I have a webcam. I run it via camorama and it takes a shot every 20 minutes. I start it and forget it, mostly using it as a rear-view mirror to tell me if someone is coming up behind me. (I have a real mirror too. I hate people sneaking up on me.)

So, I end up with lots of shots that look like this, like the wall behind me with the lights off. They're all the same shot, except they're not. They are different pictures of the same view, and thus have different sizes and different hashes. Which meant that all the standard measures of uniqueness I knew how to code are useless.

So, I did some looking and some coding, and I found a means using ImageMagick to pull some data out, which I could then use to tell ... it's not really motion-sensing, but rather whether the light is on or not. Which is enough.

#!/usr/bin/perl

use 5.010 ;
use strict ;
use warnings ;
use lib '/home/jacoby/lib' ;
use Carp ;
use Cwd 'abs_path' ;
use Data::Dumper ;
use Digest::SHA1 ;
use File::Copy ;
use HOP ':all' ;
use subs qw{ main all_files image_check } ;

my %digest ;
main '/home/jacoby/Webcam/' ;
exit ;

sub main {
    my ( $dir ) = @_ ;
    my $first = '' ;
    my $prev = '' ;
    for my $curr ( all_files $dir ) {
        my $del = 0 ;
        if ( $prev ne '' ) {
            my $check = image_check( $prev , $curr ) ;
            say join "\t" , $check , $prev , $curr ;
            $del = 1 if $check < 1 ;
            }
        if ( $del ) {
            unlink $curr ;
            next ; 
            }
        $first = $curr if $first eq '' ;
        $prev = $curr ;
        }
    } ;

sub image_check {
    my ( $pic1 , $pic2 ) = @_ ;
    my $out = qx{ compare -verbose -metric MAE $pic1 $pic2 /dev/null 2>&1 } ;
    my ( $all ) = grep m{all}mx , split m{\n} , $out ;
    $all = ( split m{\s+}mx , $all )[2] ;
    $all = int $all ;
    return $all  < 800 ? 0 : 1 ;
    }

sub all_files {
    my $dir = shift ;
    return sort { $a cmp $b } <webcam*.png> ;
    }

There are a few modules in there that I don't need, or at least that I don't need at this point in the process. There's also a qx() where I should be using Image::Magick, I know, but I just could not get that working. I tried both CPAN (which has had networking troubles for me) and apt-get (which is kinda hammered due to Lucid coming out), and neither made me happy. I figure this is another issue and I can make it better later. And better it is, because after running it, I get this:

Well, I think it's an improvement....