Cookie Notice

As far as I know, and as far as I remember, nothing in this page does anything with Cookies.
Showing posts with label graphing. Show all posts
Showing posts with label graphing. Show all posts

2012/08/28

What do I do with my weather stuff now?

It started with me wanting to know how to code with R. R is about handling and graphing data, and I didn't really have data of my own. So, what I did was find a source I liked for weather and started putting it into a MySQL database.

That source was Google, specifically the XML used for weather in the soon-to-be-closed iGoogle.

Here's that code.

#!/usr/bin/perl

# $Id: hpsetdisp.pl 11 2006-03-22 01:21:03Z yaakov $

# hpsetdisp.pl
# Connects to a JetDirect equipped HP printer and uses
# HP's control language to set the ready message on the
# LCD display.  Takes an IP address and message on the
# command line. My favorite message is "INSERT COIN".
# Keep in mind the limitations of the display when composing
# your clever verbiage.
#
# THIS PROGRAM IS PROVIDED WITH NO WARRANTY OF ANY KIND EXPRESSED OR IMPLIED
# THE AUTHOR CANNOT BE RESPONSIBLE FOR THE EFFECTS OF THIS PROGRAM
# IF YOU ARE UNCERTAIN ABOUT THE ADVISABILITY OF USING IT, DO NOT!
#
# Yaakov (http://kovaya.com/ )
#
# Modified by Garrett Hoofman
# 10/18/2007
# http://www.visionsofafar.com

# Modified by David Jacoby
# 12/10/2008
# http://varlogrant.blogspot.com/

# Modified by David Jacoby, using notify-send and tweaking Unicode
# 09/08/2008
# http://varlogrant.blogspot.com/

# Modified by David Jacoby, moving from Google to NOAA
# 08/28/2012
# http://varlogrant.blogspot.com/


use strict ;
use warnings ;
use Modern::Perl ;
use DBI ;
use Getopt::Long ;
use IO::Interactive qw{ interactive } ;
use IO::Socket ;
use XML::DOM ;
use Data::Dumper ;

use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;

use subs qw{  get_weather notify db_upload } ;

my $zip         = '' ;
my $interactive = '' ;
my $degree = '°' ; #the unicode, straight
   $degree = "\x{00b0}" ; #the unicode, specified
my $rdymsg      = "Ready" ;
my $curr_cond ;

GetOptions(
    'zipcode=s'   => \$zip ,
    ) ;

$zip !~ /\d/ and exit 0 ;
my %weather = get_weather ;
db_upload %weather ;

exit ;

########## ########## ########## ########## ########## ########## ##########
sub get_weather {
    my %output ;
    my $parser = XML::DOM::Parser->new() ;
    my $file   = 'http://www.google.com/ig/api?weather=XXXXX&hl=en' ;
    #my $zip    = shift ;
    $zip =~ s{(\d{5})}{}mx ;
    $zip = $1 ;
    $file =~ s/XXXXX/$zip/mx ;
    my $doc = $parser->parsefile( $file ) ;
    exit if ! defined $doc ;
    $output{zip}        = $zip ;
    $output{city}       = $doc->getElementsByTagName( 'city' )->item( 0 )
                        ->getAttribute('data');
    $output{temp_c}     = $doc->getElementsByTagName( 'temp_c' )->item( 0 )
                        ->getAttribute('data');
    $output{temp_f}     = $doc->getElementsByTagName( 'temp_f' )->item( 0 )
                        ->getAttribute('data');
    $output{humidity}   = $doc->getElementsByTagName( 'humidity' )->item( 0 )
                        ->getAttribute('data');
    $output{wind}       = $doc->getElementsByTagName( 'wind_condition' )->item( 0 )
                        ->getAttribute('data');
    $output{conditions} = $doc->getElementsByTagName( 'condition' )->item( 0 )
                        ->getAttribute('data') ;

    return %output ;
    }
########## ########## ########## ########## ########## ########## ##########

########## ########## ########## ########## ########## ########## ##########
sub db_upload {
    my %cond = @_ ;
    my $dbh = db_connect() ;
    for my $k ( sort keys %cond ) {
        $cond{$k} = $dbh->quote( $cond{$k} ) ;
        }
    my $fields = join ',' , qw{
        city        conditions
        humidity    temp_c
        temp_f      wind
        zip
        } ;

    my $sql ;
    my $rows ;

    $sql = qq{
    INSERT INTO weather_data ( $fields ) VALUES (
        $cond{ city } ,
        $cond{ conditions } ,
        $cond{ humidity } ,
        $cond{ temp_c } ,
        $cond{ temp_f } ,
        $cond{ wind } ,
        $cond{ zip }
        ) } ;
    #$rows = $dbh->do( $sql ) or croak $dbh->errstr;

    $sql = qq{
    INSERT INTO weather ( $fields ) VALUES (
        $cond{ city } ,
        $cond{ conditions } ,
        $cond{ humidity } ,
        $cond{ temp_c } ,
        $cond{ temp_f } ,
        $cond{ wind } ,
        $cond{ zip }
        ) } ;
    $rows = $dbh->do( $sql ) or croak $dbh->errstr;

    return ;
    }
########## ########## ########## ########## ########## ########## ##########

Problem is, Google knows I've been grabbing weather data every 10 minutes and has blocked me.

While I fill in all these, I am not pulling in weather data from multiple cities, so I can hardcode city and zip, and I couldjust drop conditions and wind. All I'm really watching is temp_f, and it's a simple conversion to get temp_c if I wanted it.

I have code that pulls from NOAA. Think I'll try to bring that in here.

2012/08/27

Weight Graphs







These are a series of generated graphs. The jagged blue line is the raw data of my more-or-less daily weight measures, and the red line is an average of the previous days' measurements. 

I don't measure every day, because I am human and therefore fallible. I take an average of the measurements of the n previous days, which means if I'm trying to get 20 and I skipped twice, I get 18. I would like to have that number graphed here, but I don't know R that well yet. Here, I'm increasing n by five, going from 5 to 30. My goal is to get to the trend line, to show I'm losing weight in total without getting caught up in the day-to-day jagged differences. After some poking, I decided to go with 20 days. Here, I've run the graph several times with different levels so I can see if there's anything better.

Clearly, the first few weeks will look similar, because a five day average will look like a 30 day average if you only have five days to work with. The trend line approximates 1 lb a week, I think. I should add that as a third line.

2010/01/14

Making Pretty Pictures

GD::Graph


R


Chart::Clicker
This should have times across the bottom. It doesn't. I don't know why.

These three images are of the same thing, graphed with different tools. I start by getting the data for these arrays out of the database:






















@number #temp_f @date
1 33 14:20
2 35 15:00
3 36 16:00
4 37 17:00
5 37 18:00
6 35 19:00
7 33 20:00
8 32 21:00
9 33 22:00
10 32 23:00
11 32 00:00
12 32 01:00
13 32 02:00
14 32 03:00
15 30 04:00
16 30 05:00
17 30 06:00
18 29 07:00
19 30 08:00
20 32 09:00


These three examples use the same basic information to draw the graph.

The database code



#!/usr/bin/perl

####################################
###### Draw Weather Graph ######
####################################

use 5.010 ;
use strict ;
use warnings ;

use Chart::Clicker;
use Chart::Clicker::Context;
use Chart::Clicker::Data::DataSet;
use Chart::Clicker::Data::Marker;
use Chart::Clicker::Data::Series;
use Chart::Clicker::Renderer::Area;
use Chart::Clicker::Renderer::Point;
use Chart::Clicker::Renderer::StackedArea;
use Geometry::Primitive::Rectangle;
use Geometry::Primitive::Circle;
use Graphics::Color::RGB;
use Graphics::Primitive::Font;
use Graphics::Primitive::Brush;

use Data::Dumper ;
use DBI ;
use Getopt::Long ;
use IO::Interactive qw{ interactive } ;

use lib '/home/jacoby/lib' ;
use MyDB 'db_connect' ;
use subs qw{
high low mean median mode range
} ;

my $dbh = db_connect() ;
my $sql ;

$sql .= 'SELECT ' ;
$sql .= 'time , ' ;
$sql .= 'temp_f ' ;
$sql .= 'FROM weather WHERE zip = "47909" ' ;
$sql .= 'AND HOUR( TIMEDIFF( SYSDATE() , time ) ) < 25 ' ;
$sql .= 'GROUP BY HOUR(time)' ;
$sql .= 'ORDER BY time' ;

my $hr = $dbh->selectall_arrayref( $sql ) or croak $dbh->errstr ;
my %tg ;

my @temp_f ;
my @date ;
my @number ;
my $count = 1 ;

for my $a ( @$hr ) {
$$a[0] = join ':' , ( split m{:} , ( split m{\s}mx , $$a[0] )[1] )[0..1];
push @date , $$a[0] ;
push @temp_f , $$a[1] ;
push @number , $count++ ;
}


Making the GD PNG



my @data = (
[@date] ,
[@temp_f]
) ;

my $graph = new GD::Graph::lines( 500,200 ) or die;
$graph->set(
x_label => 'Temperature',
y_label => 'Time',
bgclr => 'white' ,
transparent => 0 ,
title => 'Lafayette,IN Temperatures in F for the last 24 Hours',
) or die $graph->error;
$graph->set_title_font( '/home/jacoby/.fonts/trebuc.ttf' , 12 ) or die $graph->error;
$graph->set_legend_font( '/home/jacoby/.fonts/trebuc.ttf' , 8 ) or die $graph->error;
$graph->set_x_label_font( '/home/jacoby/.fonts/trebuc.ttf' , 10 ) or die $graph->error;
$graph->set_y_label_font( '/home/jacoby/.fonts/trebuc.ttf' , 10 ) or die $graph->error;
my $gd = $graph->plot(\@data) or die $graph->error;

open(IMG, '>' , '/home/jacoby/Desktop/24gd.png') or die $!;
binmode IMG;
print IMG $gd->png;




Making the R output



my $number = join ',' , @number;
my $temp_f = join ',' , @temp_f ;
my $date = join ' , ' , ( map { qq("$_") } @date );

my $r = <<"R" ;
#x <- (1:24)
x <- c( $number )
#x <- c( $date )
data <- c( $temp_f )
png(filename='/home/jacoby/Desktop/24r.png', width=500 ,height=200 )
plot(
x ,
data ,
type='l',
main='Lafayette,IN Temperatures in F for the last 24 Hours',
xlab='Time' ,
ylab='Temperature (f)'
)
#axis( 1 , at=1:48 )
dev.off()
R

open my $rh , '>' , '/home/jacoby/Desktop/24r.R' ;
say $rh $r ;

Which then has R CMD BATCH filename.R run on it to plot the graph. I could put that in as a qx{} bit, but I didn't here. Normally, yes, I would and in fact we have several that do.


Making the Chart::Clicker PNG




my $cc = Chart::Clicker->new(width => 500, height => 200, format => 'png');
my $series1 = Chart::Clicker::Data::Series->new(
keys => \@number ,
values => \@temp_f ,
);
my $dataset = Chart::Clicker::Data::DataSet->new(
series => [ $series1 ] ,
);

$cc->title->text('Lafayette,IN Temperatures in F for the last 24 Hours');
$cc->add_to_datasets($dataset);

$cc->get_context('default')->domain_axis->tick_values($series1->keys);
$cc->get_context('default')->domain_axis->tick_labels(\@date);
$cc->get_context('default')->domain_axis->tick_label_angle(1);

$cc->get_context('default')->domain_axis->fudge_amount( 0.05 );
$cc->get_context('default')->range_axis->fudge_amount( 0.1 );

my $defctx = $cc->get_context('default');
$defctx->range_axis->label( ' Temperature ' );
$defctx->domain_axis->label('Time');
$defctx->renderer->brush->width(2);
$cc->write_output('/home/jacoby/Desktop/24cc.png');

The big win is that it's graphed right. The next win is that it's all Perl, so I don't need to go back and run a plotter on the output. The last win is that it looks good.

BTW, I find it interesting that, at least on Jan 14 2010, when I do an image search on Chart::Clicker, this blog gets the top spot.

2009/12/15

A MySQL Bug? A Programmer Bug?



I have been trying to teach myself data analysis. I have been keeping some weather data in a MySQL database so I can take it out, munge it and graph it. The above example is done with the Perl module Chart::Clicker, which I am sure will take a large role in my current job. Especially if it works fast and graphs correctly.

On that topic, take a look at it. The red line represents the day's highs, the green line, the day's lows, and the purlple line the day's average temperature. Here's the code that determines what the numbers should be.

my $sql ;
$sql .= 'SELECT ' .
'cast( avg( temp_f ) as unsigned ) , ' .
'min(temp_f) , ' .
'max(temp_f) , ' .
'DATE(time) ' .
'from weather_data where zip = "47909" ' .
'and DATEDIFF( CURDATE() , DATE(time) ) < 7 ' .
'group by DATE(time)' ;

And here's the schema for this code.

CREATE TABLE weather_data (
id int(10) not null auto_increment primary key ,
time TIMESTAMP not null ,
zip varchar(5) ,
city varchar(255) ,
temp_c varchar(255) ,
temp_f varchar(255) ,
humidity varchar(255) ,
wind varchar(255) ,
conditions varchar(255)
) ;

I'll admit that it's pretty lazy. If the current conditions stretches beyond 20 characters, that's some complex weather. The main problem, I believe, is that temperature data is being saved as a string rather than a signed integer. Why it mostly shows up as error on that one day, I don't know. Here's all the temperature data for that day. (Thank you, Google Weather API.)
timetemp_ftemp_c
2009-12-10 00:00:0228-2
2009-12-10 00:10:0228-2
2009-12-10 00:30:0328-2
2009-12-10 00:40:0228-2
2009-12-10 00:50:0228-2
2009-12-10 01:00:0228-2
2009-12-10 01:10:0228-2
2009-12-10 01:30:0216-9
2009-12-10 01:40:0216-9
2009-12-10 02:00:0316-9
2009-12-10 02:10:0216-9
2009-12-10 02:20:0216-9
2009-12-10 02:40:0214-10
2009-12-10 02:50:0215-9
2009-12-10 03:00:0215-9
2009-12-10 03:10:0215-9
2009-12-10 03:20:0215-9
2009-12-10 03:30:0314-10
2009-12-10 03:40:0114-10
2009-12-10 03:50:0115-9
2009-12-10 04:00:0315-9
2009-12-10 04:10:0214-10
2009-12-10 04:20:0214-10
2009-12-10 04:30:0212-11
2009-12-10 04:40:0212-11
2009-12-10 05:00:0213-11
2009-12-10 05:10:0212-11
2009-12-10 05:20:0212-11
2009-12-10 05:30:0312-11
2009-12-10 05:40:0212-11
2009-12-10 05:50:0212-11
2009-12-10 06:00:0312-11
2009-12-10 06:10:0211-12
2009-12-10 06:20:0211-12
2009-12-10 06:30:0211-12
2009-12-10 06:40:0111-12
2009-12-10 07:00:0311-12
2009-12-10 07:10:0210-12
2009-12-10 07:20:0210-12
2009-12-10 07:30:0210-12
2009-12-10 07:40:0410-12
2009-12-10 07:50:0210-12
2009-12-10 08:00:099-13
2009-12-10 08:10:059-13
2009-12-10 08:20:069-13
2009-12-10 08:30:059-13
2009-12-10 08:40:069-13
2009-12-10 08:50:059-13
2009-12-10 09:00:068-13
2009-12-10 09:10:088-13
2009-12-10 09:20:088-13
2009-12-10 09:30:088-13
2009-12-10 09:40:138-13
2009-12-10 09:50:079-13
2009-12-10 10:00:0710-12
2009-12-10 10:10:0710-12
2009-12-10 10:30:0710-12
2009-12-10 10:40:1010-12
2009-12-10 10:50:0712-11
2009-12-10 11:00:0712-11
2009-12-10 11:10:0512-11
2009-12-10 11:20:0812-11
2009-12-10 11:40:0812-11
2009-12-10 12:00:0914-10
2009-12-10 12:30:0714-10
2009-12-10 12:50:0714-10
2009-12-10 13:00:0914-10
2009-12-10 13:10:0615-9
2009-12-10 13:20:0815-9
2009-12-10 13:30:0615-9
2009-12-10 13:50:0717-8
2009-12-10 14:00:0917-8
2009-12-10 14:10:0717-8
2009-12-10 14:20:0817-8
2009-12-10 14:40:0918-8
2009-12-10 14:50:0817-8
2009-12-10 15:20:0818-8
2009-12-10 15:30:1518-8
2009-12-10 15:40:0818-8
2009-12-10 15:50:1119-7
2009-12-10 16:00:1119-7
2009-12-10 16:10:0819-7
2009-12-10 16:20:0819-7
2009-12-10 16:30:0719-7
2009-12-10 16:40:1019-7
2009-12-10 16:50:0819-7
2009-12-10 17:00:0319-7
2009-12-10 17:10:0319-7
2009-12-10 17:30:0219-7
2009-12-10 17:40:0218-8
2009-12-10 17:50:0218-8
2009-12-10 18:00:0218-8
2009-12-10 18:10:0217-8
2009-12-10 18:20:0217-8
2009-12-10 18:30:0217-8
2009-12-10 18:40:0216-9
2009-12-10 18:50:0116-9
2009-12-10 19:00:0217-8
2009-12-10 19:10:0217-8
2009-12-10 19:20:0217-8
2009-12-10 19:40:0217-8
2009-12-10 19:50:0217-8
2009-12-10 20:00:0217-8
2009-12-10 20:20:0217-8
2009-12-10 20:30:0217-8
2009-12-10 20:40:0217-8
2009-12-10 21:00:0317-8
2009-12-10 21:10:0117-8
2009-12-10 21:20:0218-8
2009-12-10 21:30:0218-8
2009-12-10 21:40:0218-8
2009-12-10 22:00:0217-8
2009-12-10 22:10:0217-8
2009-12-10 22:20:0218-8
2009-12-10 22:30:0218-8
2009-12-10 22:40:0218-8
2009-12-10 22:50:0118-8
2009-12-10 23:00:0218-8
2009-12-10 23:10:0118-8
2009-12-10 23:20:0218-8
2009-12-10 23:30:0218-8
2009-12-10 23:40:0218-8
2009-12-10 23:50:0218-8

I could show you the temp_c graph, but that's just beyond the pale wrong, which makes sense as the temperature has been bouncing between positive and negative the whole time.

Ultimately, I got used to Perl's ability to effortlessly bounce between strings and numbers, and here I'm bitten. I can only use that data in a meaningful way if I pull it all and manipulate it in Perl, which cuts off some labor-saving tools. Let that be a lesson to you, and to me.

2009/11/24

Making Graphs with Perl and Chart::Clicker

The Graph


The Code
#!/usr/bin/perl

use 5.010 ;
use strict ;
use warnings ;
use Chart::Clicker;
use Chart::Clicker::Data::Series;
use Chart::Clicker::Data::DataSet;

my $pi = 3.14159265358979;

sub deg_to_rad { ($_[0]/180) * $pi }
sub rad_to_deg { ($_[0]/$pi) * 180 }

my @core = ( 0 .. 1_000 ) ;
my @sine ;
my @cosine ;
for my $a ( @core ) {
push @sine , sin(deg_to_rad($a)) ;
push @cosine , cos(deg_to_rad($a)) ;
}

my $cc = Chart::Clicker->new;
my $series1 = Chart::Clicker::Data::Series->new(
keys => \@core ,
values => \@sine
);
my $series2 = Chart::Clicker::Data::Series->new(
keys => \@core ,
values => \@cosine
);
my $dataset = Chart::Clicker::Data::DataSet->new(
series => [ $series1 , $series2 ],
);
$cc->add_to_datasets($dataset) ;
$cc->write_output('graph_test.png');
exit ;

There's lots of ways to do graphs. The way we do some graphs here is to do a system call to R. R is powerful, sure, but there's no good way to set up a batch of complex graphs in Perl and send them to R for graphing. There are ways, sure, but not good ways. So, I expect to do a lot more with Chart::Clicker in the future.