tag:blogger.com,1999:blog-1453328311065081862024-03-13T21:04:26.837-04:00/var/log/rantMy reasoned, well-considered thoughts on gadgets, computing, quantified self, health, open source and whatever else gets my dander up.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.comBlogger573125tag:blogger.com,1999:blog-145332831106508186.post-65164030268060380262017-11-09T12:39:00.004-05:002017-11-13T09:27:03.336-05:00Blogging ElsewhereI discovered Jekyll.<br />
<div>
<br /></div>
<div>
Jekyll allows me to write blogs in Markdown and add them with git.<br />
<br />
This is much closer to the workflow I want, because it's a very developer-centric blogging style.</div>
<div>
<br /></div>
<div>
So, right now, I am mostly blogging at <a href="https://jacoby.github.io/">htt<span id="goog_1497853446"></span><span id="goog_1497853447"></span>ps://jacoby.github.io/</a>.</div>
<div>
<br /></div>
<div>
<br /></div>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-2691579611773165342017-08-02T10:56:00.000-04:002017-08-02T10:56:05.341-04:00Considering Code DocumentationI was procrastinating about a work project, thinking about <a href="https://www.youtube.com/watch?v=Fpw3mK5uOvE" target="_blank">magnets</a>, thinking about <a href="https://www.youtube.com/watch?v=kCtwsD47nK8" target="_blank">crushers</a>, thinking about <a href="https://www.youtube.com/watch?v=k-ckechIqW0" target="_blank">thermite</a>, when I thought about my status tracker.<br />
<br />
It goes by the name <code>status.pl</code> because <code>status</code> is already a DBus utility. I made some other decisions relating to command-line interface, moving away from a cool idea I had a few years ago that hasn't aged well. I changed it to <i>record.pl</i>, which isn't similarly overloaded on my system, at least, and wrote a few flags for it, mostly for overkill. I mean, at core, it's because <a href="https://www.youtube.com/watch?v=ud3isNiB_C0&index=27&list=PLA9_Hq3zhoFxdSVDA4v9Af3iutQxLI14m" target="_blank">Len Budney convinced me my scripts needed manners</a>, especially it not doing anything when just called.<br />
<br />
So, I wrote this. All the details are tucked in Status.pm, which I am not sharing and which needs an overhaul too. All in all, I am reasonably happy with this; If there was a useless use of sub signatures award, I think this'd be a contender, but both that and postderef are used in <code>get_records.pl</code>, so it's okay. (I put no warnings because I'm running 5.26 on my desktop, but it is likely that I'll copy this to cluster machines running our 5.20 or the system 5.10 Perls, which would need the warnings.) I could and probably should go without <a href="https://metacpan.org/pod/IO::Interactive" target="_blank">IO::Interactive</a>, but still.<br />
<br />
So, my question is, beyond the user documentation that gets passed through <a href="https://metacpan.org/pod/Pod::Usage" target="_blank">Pod::Usage</a>, what documentation does this program need? I'm always at a loss for examples of what is needed. In general, programmers discount documentation; <a href="https://twitter.com/JacobyDave/status/673968596750499840" target="_blank">most editor themes I see make comments gray and muted</a>, which has always angered me, but looking at this (and yes, it's a slight case) I'm at a loss to decide what would be important to add.<br />
<br />
So, what comments would you add? What do you find wanting?<br />
<br />
(If desired, to add to this conversation, I can add the display section and the module.)<br />
<br />
<script src="https://gist.github.com/jacoby/99b1fe2b172d4b4eab84d459256776da.js"></script>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-48324935886795219652017-07-26T10:31:00.003-04:002017-07-26T10:31:48.936-04:00Using VS Code and LigaturesQuick history: I started out as a vi man, having the comical "how do I save and exit?" issues with emacs that I see lots of people complain about for vim. After college, my first job's standard editor was UltraEdit. In the lab, <a href="https://varlogrant.blogspot.com/2016/05/death-of-project.html" target="_blank">I experimented with KomodoEdit</a>, have been a Sublime Text 2 user, and right now, on both Windows and Linux, I'm using <a href="https://code.visualstudio.com/" target="_blank">Visual Studio Code</a>.<br />
<br />
As well as vim. There are occasional things VS Code can't do, or at least can't do with the packages I know about. Line sorting, for one. I like to have my modules sorted; it helps me to see if the one I want is called.<br />
<br />
Similarly, over time, I have developed a fondness for specific fonts in my editor. I cannot go through a long history of preference, but I can say that variations of Droid Sans Mono with modifications for a dotted or slashed zero do more easily distinguish them from capital O are normally what I run in terminals and editors.<br />
<br />
But, I recently saw <a href="https://twitter.com/shanselman" target="_blank">Scott Hanselman</a> blog on <a href="https://www.hanselman.com/blog/MonospacedProgrammingFontsWithLigatures.aspx" target="_blank">Monospace Programming Fonts with Ligatures</a>.<br />
<br />
What's that?<br />
<br />
Consider the code from <a href="https://varlogrant.blogspot.com/2017/07/sentimentalizing-twitter-first-pass.html" target="_blank">yesterday's post on Sentiment Analysis</a>. There are a couple places where there are skinny arrows (->), fat arrows (=>) and greater-than-and-equal signs (>=). Look specifically at line 21 for skinny arrows and greater-and-equal. Pretty, isn't it? I will also point out that line 21 also shows the dotted 0 and how easy it is to distinguish it from letters.<br />
<br />
Also look at the logical AND (&&) on like 36.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9-a57kupOrKBNYey4x_sAeCm_x6FBm4v3Sx4XLVF_5HRh19-_znKXrP3xhjpLDfXhvhFP7QorHCHDqpAQrSMNWCXbaClgDP4LYqWmqQ1RU85IY1FLH6c7HWjeUnbexoCi47-dhmb7214l/s1600/vscode_ligatures.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="998" data-original-width="840" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9-a57kupOrKBNYey4x_sAeCm_x6FBm4v3Sx4XLVF_5HRh19-_znKXrP3xhjpLDfXhvhFP7QorHCHDqpAQrSMNWCXbaClgDP4LYqWmqQ1RU85IY1FLH6c7HWjeUnbexoCi47-dhmb7214l/s640/vscode_ligatures.png" width="538" /></a></div>
<br />
These are just in-the-editor changes; the code from yesterday's post is copied from this editor.<br />
<br />
<a href="https://github.com/tonsky/FiraCode" target="_blank">The font I'm using is called Fira Code, and it's on GitHub.</a> There are others, like Monoid and Hasklig that can do this, but I like Fira Code. Within VS Code, you also have to add <code>"editor.fontLigatures": true</code> to your settings.<br />
<br />
I have it as my font for the HexChat IRC client, which supports ligatures, and Git for Windows Bash, Windows Subsystem for Linux and PowerShell terms on Windows, which don't, but the font still looks good. I noticed an issue where it swallowed the equal sign in Gnome Terminal, so there I'm back to Droid Sans Mono Slashed, but that might have been a temporary issue.<br />
<br />
Hrmm. Yesterday, I wrote on an Azure API. Today, I'm praising a Visual Studio-related editor. Tomorrow, I might get into the Windows Subsystem for Linux.<br />
<br />Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-47060892953864923602017-07-25T16:54:00.002-04:002017-07-25T16:54:09.981-04:00Sentimentalizing Twitter: First PassIt started here.<br />
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
I need a Twitter that sends all my grumpy thoughts into oblivion rather than to actual people.</div>
— Phil Sands (@PurdueCSPhil) <a href="https://twitter.com/PurdueCSPhil/status/884490097336385538">July 10, 2017</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
I couldn't leave it there.<br />
<br />
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en">
<div dir="ltr" lang="en">
You could do sentiment analysis as an alternate client or regularly after-the-fact to cut down on that. We have the technology.</div>
— Code By Java (@JacobyDave) <a href="https://twitter.com/JacobyDave/status/884494354106200064">July 10, 2017</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
I <i>can</i> write a tool that goes through all old tweets and deletes ones that don't pass criteria, but I prefer to get out ahead of the issue and leave the record as it is.<br />
<br />
And, as a later pass, one can pull a corpus, use a module like <code><a href="https://metacpan.org/pod/Algorithm::NaiveBayes" target="_blank">Algorithm::NaiveBayes</a></code> and make your own classifier, rather than using <a href="https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/quick-start" target="_blank">Microsoft Research's Text Analytics API</a> or another service. I was somewhere along that process when the hosting died, so I'm not bringing it here.<br />
<br />
I was kinda under the thrall of <i><a href="http://shop.oreilly.com/product/0636920049555.do" rel="nofollow" target="_blank">Building Maintainable Software</a></i>, or at least the first few chapters of it, so I did a few things differently in order to get their functions less than 15 lines, but the <code>send_tweet</code> function didn't get the passes it'd need, and I could probably give <code>check_status</code> some love, or perhaps even roll it into something like <code>WebService::Microsoft::TextAnalytics</code>. In the mean time, this should allow you to always tweet on the bright side of life.<br />
<br />
<pre class="perl" name="code">#!/usr/bin/env perl
use feature qw{ postderef say signatures } ;
use strict ;
use warnings ;
use utf8 ;
no warnings qw{ experimental::postderef experimental::signatures } ;
use Carp ;
use Getopt::Long ;
use IO::Interactive qw{ interactive } ;
use JSON ;
use LWP::UserAgent ;
use Net::Twitter ;
use YAML qw{ LoadFile } ;
my $options = options() ;
my $config = config() ;
if ( check_status( $options->{ status }, $config ) >= 0.5 ) {
send_tweet( $options, $config ) ;
}
else { say qq{Blocked due to negative vibes, man.} }
exit ;
sub send_tweet ( $options, $config ) {
my $twit = Net::Twitter->new(
traits => [ qw/API::RESTv1_1/ ],
consumer_key => $config->{ twitter }{ consumer_key },
consumer_secret => $config->{ twitter }{ consumer_secret },
ssl => 1,
) ;
my $tokens = $config->{ twitter }{ tokens }{ $options->{ username } } ;
if ( $tokens->{ access_token }
&& $tokens->{ access_token_secret } ) {
$twit->access_token( $tokens->{ access_token } ) ;
$twit->access_token_secret( $tokens->{ access_token_secret } ) ;
}
if ( $twit->authorized ) {
if ( $twit->update( $options->{ status } ) ) {
say { interactive } $options->{ status } ;
}
else {
say { interactive } 'FAILED TO TWEET' ;
}
}
else {
croak( "Not Authorized" ) ;
}
}
sub check_status ( $status, $config ) {
my $j = JSON->new->canonical->pretty ;
my $key = $config->{ microsoft }{ text_analytics }{ key } ;
my $id = 'tweet_' . time ;
my $object ;
push @{ $object->{ documents } },
{
language => 'EN',
text => $status,
id => $id,
} ;
my $json = $j->encode( $object ) ;
my $api = 'https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment' ;
my $agent = LWP::UserAgent->new ;
my $request = HTTP::Request->new( POST => $api ) ;
$request->header( 'Ocp-Apim-Subscription-Key' => $key ) ;
$request->content( $json ) ;
my $response = $agent->request( $request ) ;
if ( $response->is_success ) {
my $out = decode_json $response->content ;
my $doc = $out->{ documents }[ 0 ] ;
return $doc->{ score } ;
}
else {
croak( $response->status_line ) ;
}
return 1 ;
}
sub config () {
my $config ;
$config->{ twitter } = LoadFile( join '/', $ENV{ HOME }, '.twitter.cnf' ) ;
$config->{ microsoft } = LoadFile( join '/', $ENV{ HOME }, '.microsoft.yml' ) ;
return $config ;
}
sub options () {
my $options ;
GetOptions(
'help' => \$options->{ help },
'username=s' => \$options->{ username },
'status=s' => \$options->{ status },
) ;
show_usage( $options ) ;
return $options ;
}
sub show_usage ($options) {
if ( $options->{ help }
|| !$options->{ username }
|| !$options->{ status } ) {
say { interactive } <<'HELP';
Only Positive Tweets -- Does text analysis of content before tweeting
-u user Twitter screen name (required)
-s status Status to be tweeted (required)
-h help This screen
HELP
exit ;
}
}
__DATA__
.microsoft.yml looks like this
---
text_analytics:
key: GENERATED_BY_MICROSOFT
.twitter.cnf looks like this
---
consumer_key: GO_TO_DEV.TWITTER.COM
consumer_secret: GO_TO_DEV.TWITTER.COM
tokens:
your_username:
access_token: TIED_TO_YOU_AS_USER_NOT_DEV
access_token_secret:TIED_TO_YOU_AS_USER_NOT_DEV
I cover access_tokens in https://varlogrant.blogspot.com/2016/07/nettwitter-cookbook-how-to-tweet.html
</pre>
<br />Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-61273202432383848692017-07-22T16:04:00.000-04:002017-07-22T16:04:47.731-04:00The Next Thing<a href="https://varlogrant.blogspot.com/2017/07/github-to-perl-to-github-putting-your.html" target="_blank">As discussed last time</a>, I had been using my GitHub Pages space as a list to my Repos. I had been considering moving my blogging from here to ... something else, and this looked like an interesting concept.<br />
<br />
I've always developed the web with a smart server side, and I've known from the start that this makes you very vulnerable, so I do like the idea of writing markdown, committing it to the repo and having the system take care of it from there. So, <i>that's</i> a win.<br />
<br />
But, as far as I can tell, I've followed the "this is how you make an Atom feed" magic and get no magic from it, and that, more than webhooks triggering on push, is how you start putting together the social media hooks that make blogging more than writing things down in a notebook. Which is a lose.<br />
<br />
So, I'm not 100% happy with GitHub Pages and Jekyll, but the good thing is that I can write and commit from anywhere. If I used <a href="https://metacpan.org/pod/Statocles" target="_blank">Statocles</a> or another static site generator, I'd have to have that system running on whatever computer I blog from, or transfer it to there.<br />
<br />
I would guess that, if I had the whole deal running on any of my machines, some of the small things would work better, but so far, getting a setup that displays pages exactly like github.io on localhost has been less that working, And, I would've like to have this as a project page, jacoby.github.io/blog, so my personal page could be more of a landing page, but alas.<br />
<br />
And, ultimately, I do want to not have myblog.<service>.com, but rather myblog.<me>.com. Every time I think about it, I think about the tools I'd build on it, rather than the billboard for me, butDave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-14886681315760758442017-07-13T11:21:00.000-04:002017-07-13T11:21:00.949-04:00Github to Perl to Github? Putting your projects on a web pageMy projects are on GitHub: <a href="https://github.com/jacoby/">https://github.com/jacoby/</a><br /><br />I have a page on GitHub: <a href="https://jacoby.github.io/">https://jacoby.github.io/</a><br />
<br />
Early in my playing with Bootstrap, I made this as a way to begin to play with it. It is about as simple a GitHub API to LWP to Template Toolkit to Bootstrap tool as I could have written. I'm now thinking about how to make this prettier, but for now, it's what I use to generate, when I remember to redo my GitHub page. Learn and enjoy.<br />
<br />
<script src="https://gist.github.com/jacoby/5f6eb8d051f8ad3ad8b8a4f9ae1e068f.js"></script>Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-84944156518980155452017-07-07T17:49:00.003-04:002017-07-07T17:49:56.055-04:00Temperature based on Current Location for the Mobile Computer<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFdJC_38Rps0Go5uMgv9xFkOWSFecG8Eg-Ac4crzFuJYylZybD4AERECIWiEXFhlnokm3TKuGW69GetwdSX_BGakODp1Rde8qBknkjscc5DxWtHbNn8YF4xp0YtYcI0mN0szQAB-kbpL-f/s1600/My+prompt+with+temp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="390" data-original-width="652" height="380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFdJC_38Rps0Go5uMgv9xFkOWSFecG8Eg-Ac4crzFuJYylZybD4AERECIWiEXFhlnokm3TKuGW69GetwdSX_BGakODp1Rde8qBknkjscc5DxWtHbNn8YF4xp0YtYcI0mN0szQAB-kbpL-f/s640/My+prompt+with+temp.png" width="640" /></a></div>
<br />
I'm always curious about how people customize their prompt. I put name and machine in, with color-coding based on which machine, because while spend most of my time on one or two hosts, I have reason to go to several others.<br />
<br />
I work in a sub-basement, and for most of my work days, I couldn't tell you if it was summer and warm, winter and frozen, or spring and waterlogged outside, so one of the things I learned to check out is current weather information. <a href="https://varlogrant.blogspot.com/2009/02/talkin-about-weather.html" target="_blank">I used to put the temperature on the front panel of our HP printer</a>, but we've moved to Xerox.<br />
<br />
Currently, I use <a href="https://darksky.net/" rel="nofollow" target="_blank">DarkSky</a>, formerly forecast.io. I know my meteorologist friends would recommend more primary sources, but I've always found it easy to work with.<br />
<br />
I had this code talking to Redis, but I decided that it was an excuse to use Redis and this data was better suited for storing in YAML, so I rewrote it.<br />
<br />
<strong>store_temp</strong><br />
<pre class="perl" name="code">#!/usr/bin/env perl
# stores current temperature with YAML so that get_temp.pl can be used
# in the bash prompt to display current temperature
use feature qw{ say state } ;
use strict ;
use warnings ;
use Carp ;
use Data::Dumper ;
use DateTime ;
use IO::Interactive qw{ interactive } ;
use JSON ;
use LWP::UserAgent ;
use YAML::XS qw{ DumpFile LoadFile } ;
my $config = config() ;
my $url
= 'https://api.darksky.net/forecast/'
. $config->{apikey} . '/'
. ( join ',', map { $config->{$_} } qw{ latitude longitude } ) ;
my $agent = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 } ) ;
my $response = $agent->get($url) ;
if ( $response->is_success ) {
my $now = DateTime->now()->set_time_zone('America/New_York')->datetime() ;
my $content = $response->content ;
my $forecast = decode_json $content ;
my $current = $forecast->{currently} ;
my $temp_f = int $current->{temperature} ;
store( $now, $temp_f ) ;
}
else {
say $response->status_line ;
}
exit ;
<pre class="perl" name="code"># ======================================================================
</pre>
sub store {
my ( $time, $temp ) = @_ ;
say {interactive} qq{Current Time: $time} ;
say {interactive} qq{Current Temperature: $temp} ;
my $data_file = $ENV{HOME} . '/.temp.yaml' ;
my $obj = {
curr_time => $time,
curr_temp => $temp,
} ;
DumpFile( $data_file, $obj ) ;
}
# ======================================================================
# Reads configuration data from YAML file. Dies if no valid config file
# if no other value is given, it will choose current
#
# Shows I need to put this into a module
sub config {
my $config_file = $ENV{HOME} . '/.forecast.yaml' ;
my $output = {} ;
if ( defined $config_file && -f $config_file ) {
my $output = LoadFile($config_file) ;
$output->{current} = 1 ;
return $output ;
}
croak('No Config File') ;
}
</pre>
<br />
And this is the code that reads the YAML and prints it, nice and short and ready to be called in PS1.<br />
<br />
<strong>get_temp</strong><br />
<pre class="perl" name="code">#!/usr/bin/env perl
# retrieves the current temperature from YAML to be used in the bash prompt
use feature qw{ say state unicode_eval unicode_strings } ;
use strict ;
use warnings ;
use utf8 ;
binmode STDOUT, ':utf8' ;
use Carp ;
use Data::Dumper ;
use YAML::XS qw{ LoadFile } ;
my $data_file = $ENV{HOME} . '/.temp.yaml' ;
my $output = {} ;
if ( defined $data_file && -f $data_file ) {
my $output = LoadFile($data_file) ;
print $output->{curr_temp} . '°F' || '' ;
exit ;
}
croak('No Temperature File') ;</pre>
<br />
I thought I put a date-diff in there. I wanted to be able say 'Old Data' if the update time was too long ago. I should change that.<br />
<br />
I should really put the config files in __DATA__ for show, because it will show that the location is hard-coded. For a desktop or server, that makes sense; it can only go as far as the power plug stretches. But, for other reasons, I adapted my bash prompt on my Linux laptop, and I recently took it to another state, so I'm thinking more and more that I need to add a step, to look up where I am before I check the temperature.<br />
<br />
<strong>store_geo_temp</strong><br />
<pre class="perl" name="code">#!/usr/bin/env perl
# Determines current location based on IP address using Google
# Geolocation, finds current temperature via the DarkSky API
# and stores it into a YAML file, so that get_temp.pl can be
# in the bash prompt to display current local temperature.
use feature qw{ say state } ;
use strict ;
use warnings ;
use utf8 ;
use Carp ;
use Data::Dumper ;
use DateTime ;
use IO::Interactive qw{ interactive } ;
use JSON::XS ;
use YAML::XS qw{ DumpFile LoadFile } ;
use lib $ENV{HOME} . '/lib' ;
use GoogleGeo ;
my $json = JSON::XS->new->pretty->canonical ;
my $config = config() ;
my $location = geolocate( $config->{geolocate} ) ;
croak 'No Location Data' unless $location->{lat} ;
my $forecast = get_forecast( $config, $location ) ;
croak 'No Location Data' unless $forecast->{currently} ;
say {interactive} $json->encode($location) ;
say {interactive} $json->encode($forecast) ;
my $now = DateTime->now()->set_time_zone('America/New_York')->datetime() ;
my $current = $forecast->{currently} ;
my $temp_f = int $current->{temperature} ;
store( $now, $temp_f ) ;
exit ;
# ======================================================================
# Reads configuration data from YAML files. Dies if no valid config files
sub config {
my $geofile = $ENV{HOME} . '/.googlegeo.yaml' ;
croak 'no Geolocation config' unless -f $geofile ;
my $keys = LoadFile($geofile) ;
my $forecastfile = $ENV{HOME} . '/.forecast.yaml' ;
croak 'no forecast config' unless -f $forecastfile ;
my $fkeys = LoadFile($forecastfile) ;
$keys->{forecast} = $fkeys->{apikey} ;
croak 'No forecast key' unless $keys->{forecast} ;
croak 'No forecast key' unless $keys->{geolocate} ;
return $keys ;
}
# ======================================================================
# Takes the config for the API keys and the location, giving us lat and lng
# returns the forecast object or an empty hash if failing
sub get_forecast {
my ( $config, $location ) = @_ ;
my $url
= 'https://api.darksky.net/forecast/'
. $config->{forecast} . '/'
. ( join ',', map { $location->{$_} } qw{ lat lng } ) ;
my $agent = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 } ) ;
my $response = $agent->get($url) ;
if ( $response->is_success ) {
my $content = $response->content ;
my $forecast = decode_json $content ;
return $forecast ;
}
return {} ;
}
<pre class="perl" name="code"># ======================================================================
</pre>
sub store {
my ( $time, $temp ) = @_ ;
say {interactive} qq{Current Time: $time} ;
say {interactive} qq{Current Temperature: $temp} ;
my $data_file = $ENV{HOME} . '/.temp.yaml' ;
my $obj = {
curr_time => $time,
curr_temp => $temp,
} ;
DumpFile( $data_file, $obj ) ;
}
</pre>
<br />
A few things I want to point out here. First off, you could write this with Getopt::Long and explicit quiet and verbose flags, but Perl and IO::Interactive allow me to make this context-specific and implicit. If I run it myself, interactively, I am trying to diagnose issues, and that's when <code>say {interactive}</code> works. If I run it in crontab, then it runs silently, and I don't get an inbox filled with false negatives from crontab. This corresponds to my personal preferences; If I was to release this to CPAN, I would likely make these things controlled by flags, and perhaps allow latitude, longitude and perhaps API keys to be put in that way.<br />
<br />
But, of course, you should not get in the habit, because then your keys show up in the process table. It's okay if you're the only user, but not best practice.<br />
<br />
This is the part that's interesting. I need to make it better/strong/faster/cooler before I put it on CPAN, maybe something like Google::Geolocation or the like. Will have to read some existing Google-pointing modules on <a href="https://metacpan.org/" target="_blank">MetaCPAN</a> before committing. <a href="https://metacpan.org/pod/Geo::Google" rel="nofollow" target="_blank">Geo::Google</a> looks promising, but it doesn't do much with "Where am I now?" work, which is exactly what I need here.<br />
<br />
Google's Geolocation API works better when you can point to access points and cell towers, but that's diving deeper than I need; the weather will be more-or-less the same across the widest accuracy variation I could expect.<br />
<br />
<strong>GoogleGeo</strong><br />
<pre class="perl" name="code">package GoogleGeo ;
# interfaces with Google Geolcation API
# https://developers.google.com/maps/documentation/geolocation/intro
use feature qw{say} ;
use strict ;
use warnings ;
use Carp ;
use Data::Dumper ;
use Exporter qw(import) ;
use Getopt::Long ;
use JSON::XS ;
use LWP::Protocol::https ;
use LWP::UserAgent ;
our @EXPORT = qw{
geocode
geolocate
} ;
my $json = JSON::XS->new->pretty ;
my $agent = LWP::UserAgent->new ;
sub geocode {
my ($Google_API_key,$obj) = @_ ;
croak unless defined $Google_API_key ;
my $url = 'https://maps.googleapis.com/maps/api/geocode/json?key='
. $Google_API_key ;
my $latlng = join ',', $obj->{lat}, $obj->{lng} ;
$url .= '&latlng=' . $latlng ;
my $object = { latlng => $latlng } ;
my $r = $agent->post($url) ;
if ( $r->is_success ) {
my $j = $r->content ;
my $o = $json->decode($j) ;
return $o ;
}
return {} ;
}
sub geolocate {
my ($Google_API_key) = @_ ;
my $url = 'https://www.googleapis.com/geolocation/v1/geolocate?key='
. $Google_API_key ;
my $object = {} ;
my $r = $agent->post( $url, $object ) ;
if ( $r->is_success ) {
my $j = $r->content ;
my $o = $json->decode($j) ;
return {
lat => $o->{location}{lat},
lng => $o->{location}{lng},
acc => $o->{accuracy},
} ;
}
return {} ;
}
'here' ;
</pre>
<br />
If this has been helpful or interesting to you, please tell me so in the comments.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-76131124545332426342017-07-06T16:24:00.000-04:002017-07-06T16:24:14.022-04:00Working Through Limitations: The Perl Conference 2017 Day 3<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1VbpuR9wa_HQXVD5TkAiRhcEYqu4cIjjfw8TWPkN8SlYQct7BFNsJQ2Ikn09ZATS_f3V87Z2yPZN8slPF4ctuRt__7eM9D2eVB12ro1S2Z1bWrKXajl1g5PPfqzH7pgC4T4rNKev_avfl/s1600/IMG_20170620_131946.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1VbpuR9wa_HQXVD5TkAiRhcEYqu4cIjjfw8TWPkN8SlYQct7BFNsJQ2Ikn09ZATS_f3V87Z2yPZN8slPF4ctuRt__7eM9D2eVB12ro1S2Z1bWrKXajl1g5PPfqzH7pgC4T4rNKev_avfl/s640/IMG_20170620_131946.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A Delorean with a Perl-powered center column and the cutest little Flux Capacitor on the dashboard. Oh, the wonders you can see at a developer conference.</td></tr>
</tbody></table>
"A man's got to know his limitations."<br />
<br />
That's a line from Dirty Harry Callahan in <i>Magnum Force</i>, but it really described my planning for the Perl Conference. Once the calendar was up, I went in, first and foremost thinking "What are skills I need to learn?"<br />
<br />
One crucial skill is version control. It's difficult to add to my main workflow, as I develop in production. (I live in fear.) But I'm increasingly adding it to my side projects. It is especially part of the process for maintaining the site for <a href="http://purdue.pl/" target="_blank">Purdue Perl Mongers</a>, as well as aspects of <a href="https://www.meetup.com/hacklafayette/" target="_blank">HackLafayette</a>, but beyond certain basics, I just didn't know much about how to use Git and version control to improve my projects. I learned how <a href="http://cpantesters.org/" target="_blank">CPAN Testers</a> tests your code on many platforms after you upload to CPAN, and how <a href="https://travis-ci.org/" target="_blank">Travis-CL</a> and <a href="https://www.appveyor.com/" target="_blank">Appveyor</a> test against Linux, macOS and Windows after pushing to GitHub, but how track changes, align issues with branches, etc., are all new to me. So, I started Wednesday with <b>Genehack and Logs Are Magic: Why Git Workflows and Commit Structure Should Matter To You</b>. (<a href="https://speakerdeck.com/genehack/logs-are-magic-why-git-workflows-and-commit-structure-should-matter-to-you" rel="nofollow" target="_blank">Slides</a>) I fully expect to crib ideas and aliases from this talk for some time to come.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/n_WGiS8fm8s" width="640"></iframe>
<br />
<br />
There was a talk at a local software group featuring a project leader from Amazon on the technology involved with Alexa, which involved a lot of how this works, going from speech-to-text to tokenization and identifying of crucial keywords -- "Alexa, have Domino's order me a Pizza" ultimately boiling down to "Domino's" and "Pizza" -- and proceeding from there. It gave a sense of how Amazon is taking several hard problems and turning them into consumer tools.<br />
<br />
What came very late in the talk is how to interface my systems with Amazon's <a href="http://iotpodcast.com/2017/02/episode-98-science-fiction-prepared-me-for-spying-tvs/" rel="nofollow" target="_blank">"Talking Donkey"</a>, and I had a few conversations where we talked about starting the day with "Alexa, what's fresh hell is this?" and getting back a list of systems that broke overnight, but I lacked a strong idea of what is needed to make my systems interact with the Alexa system.<br />
<br />
And <a href="https://www.amazon.com/dp/B015TJD0Y4" target="_blank">an Echo</a>.<br />
<br />
But, thankfully, <b>Jason Terry's Amazon, Alexa and Perl</b> talk covered this, albeit more in the "Turn my smart lights on" sense than in the "Tell me what my work systems are doing" sense. Still, very much something I had been interested in.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/Q7hGl_WKsDA" width="640"></iframe>
<br />
<br />
But, as I implied, Amazon does a <i>lot</i> of heavy lifting with Alexa, getting it down to GETs and and POSTs against your API. If you're running this from home, where you have a consistent wired connection, this works. But, if you're running this, for example, in your car, you need it to be small, easy, and self-contained. <b>Chris Prather</b> decided to <b>MAKE New Friends</b> with a Raspberry Pi 3. This was a case of Conference-Driven Development, and he didn't have it ready to demonstrate at this time.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/KmoUk3a9fH4" width="640"></iframe><br />
<br />
I've been trying to move my lab to the New Hotness over time, and because I will have to tie things back to existing systems, I have avoided learning too much in order to make it work. <b>Joel Berger</b> presented <b>Vue.js, Mojolicious, and PostgreSQL chat in less than 50 lines</b>. I've heard good things about <a href="https://vuejs.org/">Vue.js</a>, which I heard from the project head was the parts he needed from Angular without the stuff he didn't use. (<a href="https://changelog.com/rfc/12">RFC Podcast</a>) I use jQuery and Template and not much more, so the "only what you need" aspect sounds good to me. I fully expect to bug Joel on Twitter and IRC about this and other things I need to do with <a href="http://mojolicious.org/">Mojolicious</a> over the coming months. (<a href="https://jberger.github.io/VueChat/#/">Slides</a>)<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/dDH10srLgVc" width="640"></iframe><br />
<br />
But, of course, not every talk needs to speak directly to my short-term needs. <b>Stevan Little</b> has been trying to add an object system to the Perl 5 for quite some time, and in <b>Hold My Beer and Watch This</b>, he talks about <a href="https://metacpan.org/pod/Moxie">Moxie</a>, his latest attempt.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/w5U7eoeuO90" width="640"></iframe><br />
<br />
OK, the Moxie talk was in the same room as the Mojolicious talk and this next one. I <i>could</i> have gone to one of the others, but I decided against. Oh well. I would put "do more object-oriented development" as a thing to learn, so I was glad to hear it.<br />
<br />
The last full talk was <b>The Variable Crimes We Commit Against JavaScript by Julka Grodel</b>. I would say that I code Javascript half as much as I code Perl, but twice as much as I code in other languages. I knew certain parts, like how <i>let</i> gives lexical scoping like Perl's <i>my</i>. I had heard about <i>let</i> from my JS Twitter list, as well as from Matt Trout's talk, but there were certainly things I didn't know.<br />
<br />
And, actually, things I still don't. I had technical difficulties with my laptop, and if I could've worked it out that day, I would've tried to set up a lightning talk. Alas, not only did it not work -- in the end, I swapped out the WiFi, and I think if I switched back, it'd be sane again -- I missed a lot of her talk about arrow functions, which piqued the interest of Perl's functional programming guru, Mark Jason Dominus. (In a not-surprising addition to my limitations, I still haven't gotten to the end of <i>Higher Order Perl</i>.) Anyway, I believe that, after I finish this post, I will go back and watch this again.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/1aLoq9XzufU" width="640"></iframe><br />
<br />
After this, there were Lightning Talks, interspersed with ads for various Perl Mongers groups, and urges for you to start a Perl Mongers group. I am likely going to do another post, with these talks from all three days, but to end this one, I'll show off one where M. Allan Noah talks about the <a href="https://gitlab.com/sane-project" rel="nofollow" target="_blank">SANE Project</a> and how he reverse engineers scanner output to allow him to support scanners without vendor input.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" src="https://www.youtube.com/embed/pJ60vVOKVr0" width="640"></iframe><br />
<br />
Lack of documentation is a limitation, but knowing the limitation does not mean you have to stop, and lack of documentation will not stop him. We should all draw inspiration from that.<br />
<br />
Now that I'm two weeks past the end, most of the talks are up, and I would like to commend the organizers. Conference organization is hard, and this venue made it harder. The US Patent and Trademark Office is a great place, but there are aspects that they as a venue wanted secured and I would've preferred to be more open, but it was a beautiful venue and I be glad to return.<br />
<br />
But, the thing I'd like to commend them most on is the high quality and fast turnaround on talk videos. The audio is good, the video is clear and well-lit, and the slides take precedence over the speaker in framing. It's everything I want in a conference video.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-80063060785845494992017-06-29T18:03:00.002-04:002017-07-03T13:54:49.280-04:00Three Little Words: The Perl Conference 2017 Day 2I lost the bubble.<br />
<br />
<a href="https://varlogrant.blogspot.com/2017/06/feeling-tipsy-perl-conference-2017-day-1.html" rel="nofollow" target="_blank">I wrote a post at the end of the first day of the Perl Conference</a>, intending to do the same for the second and third, but I went to visit family and play with their puppies on Tuesday night, and puppies are better than blogging.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPEcJp4kqllSbSm-esPDJgLCSpUZbk-rZUgjRuK1IviCFWJc3uaVx2qjpzcon7_CxifAD9nvUCflAP7f4MZZabqMU2HXdaQbGHW9_1UAVUEhZgn1RrrK_yuPoi-TiONRhE7lBz9LyQA7Yt/s1600/IMG_20170620_203036.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1185" data-original-width="1600" height="472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPEcJp4kqllSbSm-esPDJgLCSpUZbk-rZUgjRuK1IviCFWJc3uaVx2qjpzcon7_CxifAD9nvUCflAP7f4MZZabqMU2HXdaQbGHW9_1UAVUEhZgn1RrrK_yuPoi-TiONRhE7lBz9LyQA7Yt/s640/IMG_20170620_203036.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Puppies!</td></tr>
</tbody></table>
The beginning of the third day, my laptop hit a weird issue where it couldn't enable the wireless NIC. I fixed it at home, once I had access to precision screwdrivers again, but this meant that I couldn't blog the last day, either.<br />
<br />
But waiting affords me the opportunity to add videos of the talks I attended. So, yay.<br />
<br />
I started Day 2 with <b>Sam Batschelet's Dancing In The Cloud</b>, which was not what I was expecting, being more a deep dive in another domain. I was hoping for an introduction to using the Dancer framework on cloud providers. It was interesting, and I now know what <a href="https://kubernetes.io/" rel="nofollow" target="_blank">kubernetes</a> does and that etcd exists, but this is far enough away from where I live that I don't know if it'll give me much more than buzzword recognition when I next attend a technical conference.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/HG0eOWESV0U" width="560"></iframe>
<br />
Again, I would like to reiterate that it was deep for me. Good talk, good speaker, but I was the wrong audience. <i>Read the talk descriptions, not just the titles!</i><br />
<br />
This was followed by <b>Chad Granum on Test2</b>, which is not yet on YouTube. I came to TPC wanting to up my testing game, and I've been looking through Andy Lester's modules and brian d foy's Stack Overflow posts trying to get into the mindset. This talk is a long-form walk through the diffs between the old and new test frameworks, and by showing how to test using <b><a href="https://metacpan.org/pod/Test2" target="_blank">Test2</a></b>, it showed a lot about how to test in general. Once the talk is up, I am sure I will go through it again and again, trying to get my head around it.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/wGTZCaU_z5U" width="560"></iframe>
<br />
But just because we don't have this talk on YouTube, that doesn't mean we don't have Chad talking about testing.<br />
<br />
My next talk was <b>Adventures in Failure: Error handling culture across languages by Andrew Grangaard</b>, which is another point of growth for me. All too often, I just let errors fall, so knowing how they're commonly done is a good thing, well presented.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/VbObDEH-jKY" width="560"></iframe>
<br />
Damian Conway was the keynote speaker, and his talk was called Three Little Words. Those are, of course, <b>I ❤ Perl.</b><br />
<div>
</div>
<br />
Also, <b>class has method</b>. Conway proceeded to implement Perl6-style OOP with <b><a href="https://metacpan.org/pod/Dios" target="_blank">Dios</a></b>, or Declarative Inside-Out Syntax, which needed <b><a href="https://metacpan.org/pod/Keyword::Declare" target="_blank">Keyword::Declare</a></b> to enable the creation of new keywords in Perl 5. To get this going, he needed to more quickly parse Perl, so along comes <b><a href="https://metacpan.org/pod/PPR" target="_blank">PPR</a></b>, the Pattern-based Perl Recognizer. It was in this talk that the previous Lightning Talk from Chad, including his module <b><a href="https://metacpan.org/pod/distribution/Test2-Plugin-SourceDiag/lib/Test2/Plugin/SourceDiag.pm" target="_blank">Test2::Plugin::Source</a></b>, comes from.<br />
<br />
The <b>Lightning Talks</b> were good, but the one that comes up as a good one to put here is <b>Len Budney's Manners for Perl Scripts</b>. This makes me want to look into <b><a href="https://metacpan.org/pod/CLI::Startup" target="_blank">CLI::Startup</a></b>.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/ud3isNiB_C0" width="560"></iframe>
<br />
I hope to get to Day 3 and maybe another post just on Lightning Talks soon. An hour's dive into topics won't get you too deep, but an hours' lightning talks will will open up a number of possibilities for you to follow.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-77766077956406257402017-06-27T09:36:00.000-04:002017-06-27T10:31:36.723-04:00Reverse Engineering Google Maps at Highway Speed<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7QkKjdfC871P5iHDimiyvL330C7OEUdBxIndiho0RD_SkJcNoHEDYqEVwrIoA7UW_xPjMdi7NLIPWLjGcr_sYsBBIRAG6MnCXrM9unNkvYNhCEK3DO4IHlSsIa4ZFo4Es55Wka_4jko6t/s1600/Screenshot_20170623-154557-01.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1080" data-original-width="1080" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7QkKjdfC871P5iHDimiyvL330C7OEUdBxIndiho0RD_SkJcNoHEDYqEVwrIoA7UW_xPjMdi7NLIPWLjGcr_sYsBBIRAG6MnCXrM9unNkvYNhCEK3DO4IHlSsIa4ZFo4Es55Wka_4jko6t/s640/Screenshot_20170623-154557-01.jpeg" width="640" /></a></div>
<br />
<br />
I was on I-70 in Maryland on Sunday, going to Alexandria, Virginia, along with a lot of others. I was using Google Maps for navigation. When I could look down, the route was looking red, indicating congestion and delay. Eventually, Google said, "We have an alternate route that will save you a half hour. Want to take it?"</div>
<br />
<div>
Of course I said yes. So I took the next exit, went down some rural highways, through a small town and almost down someone's driveway, it seemed, and back onto I-70. I called it a win.</div>
<br />
<div>
The Friday after, on my way home, it rained all the way through Maryland, West Virginia, Pennsylvania and Ohio, and well into Indiana. It occasionally rained hard enough that I started seeing other drivers turning on their hazard lights for others to see them, and I followed suit.</div>
<br />
<div>
A truck had crashed in western Ohio, closing westbound lanes, and Google told me it would add an hour delay, but when it routed me through five miles of county roads to the next on-ramp, it was closed. I knew it -- I saw the chain across the road -- but Google didn't, so I alone, without a line of other vehicles, bird-dogged a route to the next on-ramp with Google urging me to make a u-turn every few hundred feet.</div>
<br />
<div>
This made me think about what's actually going on inside the navigation feature of Google Maps. It starts in graph theory, establishing the US road system as a directed graph with weighted edges, showing Interstates as preferable to highways to county roads and side streets, and using <a href="https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm" target="_blank">Dijykstra's shortest path algorithm</a> to find the best route.</div>
<br />
<div>
In graph theory, everything is either a node or an edge. In this case, a node could be the intersection of 6th and Main, or the fork where I-70 splits off to I-270. The edges would be the road between 5th and 6th on Main St., or the long path between exits on the Interstate. In other uses of graph theory, the nodes are most important -- Facebook Users being nodes and their friendships being edges in Facebook's Social Graph -- but here, the information about the edges is the important part.</div>
<br />
<div>
This is similar to how we used to do it, looking at <u>a</u> paper map and preferring Interstate for long-haul routes, judging this road or that road as preferable due to scenery or a hundred other criteria, dropping to surface streets only to get through the final few miles. But, Google knows this road gets gridlocked at rush hour and that one is under construction, which you can't tell from your ten-year-old road atlas, and has a huge body of historical data that allows it to present alternate routes and the estimated time difference between.</div>
<br />
<div>
The increasing amount of data helps it properly weigh edges and make connections. Early in my time with Maps, it suggested I go from Lafayette to Ft Wayne through Indianapolis, which would actually add an hour to the trip, because it had weighted the Interstate so highly. Now, it properly suggests two east-west routes and avoids the southern detour entirely. Similarly, an intersection near work is right-turn only, and it took some time for Maps to not suggest a left turn.</div>
<br />
<div>
Maps re-calculates the route often, which is why, after a time off its path, it still says "fastest way is behind you; turn around and go back". When you're just stopping for gas, it's a little annoying, but when you see that the highway department has blocked the suggested route, you wish it would just shut up and get with the program. I'd have to take more trips where I'm not the driver to really tell, but I would guess it re-runs shortest-path several times a minute. My guess is there are waypoints, places along the route between here and there, so Maps tries to find the shortest path between them, rather than rethinking every turn in your 600-mile journey, which makes this faster and more predictable.</div>
<br />
<div>
Maps has a lot of data for each section of road, showing how many drivers use it and their average speed, as well as the posted speed limit. If the drivers are going slower than average and slower than the speed limit, that indicates there is a problem. In the Ohio case, the Department of Transportation must have reported that the reason was an accident, but to paraphrase <i>Scream</i>, causes are incidental. What's important is knowing where things get back to normal, and if cars aren't there, then Maps has no way of knowing where that is and what on-ramp is open. This is guesswork of the highest order, but in a week where my every movement was guided by Maps' algorithms, this was the only point of failure.</div>
<br />
<div>
When I took that detour in Maryland, I was not alone; I counted at least eight vehicles taking that route along with me. Google offered me the choice, but this can't have been an option for every driver on a backed-up highway. I think there must have been a process of A-B testing, where some were rerouted and some were kept on the main road, and it used this information to decide where to send drivers later.</div>
<br />
<div>
I don't often take these long drives, so it may be a year or two before I'm so fully in the hands of Google Maps and on such a dynamic journey, but I expect the experience to be even better.</div>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-61956940689483136262017-06-19T22:47:00.002-04:002017-06-29T18:02:19.311-04:00Feeling "Tipsy": The Perl Conference 2017 Day 1A few lessons learned before I dive into the talks I attended and what I gained from them:<br />
<br />
<ul>
<li>I had figured that aspects of my Every Day Carry would not be appreciated in a government building, so I left my multitool off my belt, but, even after I had put everything I could think of into the tray, I had still made the machines go beep, causing the guards to pull out the wand. Tomorrow, pack lighter.</li>
<li>I take notes on a paper notebook, but I used my phone and tablet to connect to others via IRC and Twitter, as well as keeping track of my calendar, and I ran through my batteries before the end of the talks. In part, it was switching between the local WiFi and the cellular network, doing well with neither, but I'm not convinced it'd wouldn't be drained regardless. At lunch, I need to find a place to charge. I often come with a power strip for just this purpose, too.</li>
<li>I didn't break out the laptop once. If I don't use it more, I should just leave it and have less to carry.</li>
</ul>
<div>
Hopefully, I will remember this and come prepared for the next one.</div>
<div>
<br /></div>
<div>
Now, to the talks:</div>
<div>
<br /></div>
<div>
I started the day with <b>MetaCPAN, the Grand Tour</b>. I had been convinced earlier that MetaCPAN is better than CPAN for looking into modules, but there's more, including modules I am going to have start using to search CPAN. </div>
<div>
<br /></div>
<div>
Graham Knop's <b>Continuous Integration for CPAN</b> followed. I had been aware of Travis-CI, and had become aware of AppVeyor recently, but the tooling available in Perl to work with these was less familiar to me. I was unaware that you can specify Linux or OSX in Travis, This was something I was thinking and asking questions of other developers about. I have issues on FreeBSD, which I'm told is something that Gitlab-CI can help me with, but somehow, I doubt I can connect Github to Gitlab, but I could be wrong.</div>
<div>
<br /></div>
<div>
Steven Lembark had much more with <b>Dockerizing CPAN Testers: Running an Isolated Test Site</b> than I could fit into my head, and I think I'll have to go back to the tape once it's available, but I think it's a useful addition to the world.</div>
<div>
<br /></div>
<div>
After lunch, I went to Joel Berger's <b>Variables, Scoping and Namespaces</b>, which he set as a talk for beginners. He went so far as to suggest more established developers go elsewhere. Since I never thought I learned all of Perl when I was learning, it was very much a lot of things I did already know, but a little "So <i>that's</i> why we do that", some more "Oooh. I forgot about that", and one weird trick that explains how to mock functions for tests.</div>
<div>
<br /></div>
<div>
(That, fundamentally, is my big item to work on as a developer. Testing.)</div>
<div>
<br /></div>
<div>
After this, I attended Matt S. Trout's <b>ES6: Almost an acceptable Perl 5?</b> and it gave me a sense of treating Javascript like Perl, but since I don't code Perl like Matt does, I probably won't code ES6 like Matt does. My notes peter out about halfway through, but they do include several gems, such as lodash, that might improve the code I do write.</div>
<div>
<br /></div>
<div>
Following this is Lightning Talks, which has a bunch of interesting points, going from "Write for OpenSource.com" to "Try learning Dart" to "Creating Test JSON from JSON Schemas" to "Using Instapaper, IFTTT and Buffer to tweet what you read" to the Alien modules, which I almost understand now. Or maybe not. Certainly, though, I'd be installing and trying Dave Rolsky's <a href="https://metacpan.org/pod/RTx::ToGitHub" rel="nofollow" target="_blank">RTx-toGitHub</a> right now if I wasn't so tired.</div>
<div>
<br /></div>
<div>
Finally, Sawyer X talked about Perl 5.26 and the changes that came and the changes that are coming. The thing that comes to mind first is that things that have been deprecated since 5.0 are finally being pulled. I understand the critics who think that removing . from @INC is insignificant, but I am still for it. I also like that Perl recognizes unhandled merges and dies now. </div>
<div>
<br /></div>
<div>
Tomorrow, I will be learning about Dancer, Test2 and more with @INC, and visiting with family after</div>
<div>
<br /></div>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-40832154968267518532017-06-02T15:03:00.002-04:002017-06-02T15:03:32.905-04:00Tracking your Old Tabs in Chrome over TimeStarting "Has <i>this</i> ever happened to you?" is a very informercial way to start, but it's where my brain has left me: Working on your computer and suddenly, something happens. Blue screen. Kernel panic. Browser crash. Whatever. Restart things, open Chrome and click "Restore Tabs" and it ... does nothing.<br />
<br />
Or does something, but not enough.<br />
<br />
A user's list of open browser tabs and windows is a map of that user's interests. For me, there's usually 6-20 open windows with 3-12 tabs covering a number of topics that, while interesting to you, are not directly applicable right now.<br />
<br />
And Chrome is of no help these days. As Neil Young said "It's all the same song", Google believes, between phones, tablets, laptops, desktops — heck, the talking-donkey called Google Home, for all I know — that it's all the same browser. Pages you saw recently, no matter what device, are at the top of <code>chrome://history</code>, and the tabs that you kept around for when you get back to that topic are way dropped out, and ironically, the tabs you looked at, rejected and closed are right at the top.<br />
<br />
Last night, I killed lots of tabs to enhance browser stability. Today, looking for an image, I screwed up my machine, and so I rebooted. It's Linux and that's bad Linux admin work, but it wasn't a hill I wanted to die on, and when I got back on, the six windows and maybe 20 tabs total were gone. There were certain givens (Tweetdeck, Chrome) and certain trashables (work tabs for tasks completed the day before) but the "I was gonna get back to that" windows are gone, and the pages they held are deep deep deep in the communal history behind today's web comics and headline links.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjf_lX-MpZGZ3yHMRyGt6NLrKonb44-cPzioYdk3JaganUeu50Ghh802FFoFx02qSOFKTbiXkHQ0AbXumxaUftONKblEIdsQeoPnnTYteaogW4FYv1bpkcJLYslCza4WoXbERC38xe3D6Cl/s1600/facepalm.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjf_lX-MpZGZ3yHMRyGt6NLrKonb44-cPzioYdk3JaganUeu50Ghh802FFoFx02qSOFKTbiXkHQ0AbXumxaUftONKblEIdsQeoPnnTYteaogW4FYv1bpkcJLYslCza4WoXbERC38xe3D6Cl/s400/facepalm.jpg" width="400" /></a></div>
<br />
I cannot get those pages and the plans I built on them back. But, I can start to get my house in order to keep it from happening again. And that starts with storing them.<br />
<br />
<a href="http://www.chromium.org/user-experience/user-data-directory" rel="nofollow" target="_blank">This page shows the default location for Chrome user data.</a> <a href="http://lifehacker.com/how-can-i-restore-closed-tabs-after-accidentally-quitti-1452314285" rel="nofollow" target="_blank">Lifehacker has a how-to on Restoring tabs</a>, detailing what files hold what you need and how to force Chrome into Recovery Mode. the key things are in <code>Current Session</code>, <code>Current Tabs</code>, <code>Last Session</code> and <code>Last Tabs</code>. These files are SNSS format, which could be better (I've found <a href="https://github.com/JRBANCEL/Chromagnon/tree/SNSS" target="_blank">a Github repo with an SNSS parser in Python</a>, but haven't started working with it), but they respond to <code>strings</code>, so you have something to work on without the parser. <br />
<br />
<pre class="perl" name="code">#!/usr/bin/env perl
use feature qw{ say state } ;
use strict ;
use warnings ;
use utf8 ;
use DateTime ;
use File::Copy ;
use File::Path qw{ make_path } ;
use IO::Interactive qw{ interactive } ;
# program to back up chrome tabs for easy restore should things go bad
# add to your crontab with something like
# @hourly ~/bin/chrome_tab_backup.pl
# T
#
my $now = DateTime->now()
->set_time_zone('America/Indiana/Indianapolis')
->set( second => 0 ) ;
my $date = $now->ymd('/') ;
my $time = $now->hms('-') ;
my $source = join '/', $ENV{HOME}, qw{ .config chromium Default } ;
my $target = join '/', $ENV{HOME}, '.chrome_backup', $date, $time ;
say $source ;
say $target ;
if ( !-d $target ) {
my @dirs = make_path($target) ;
}
chdir $source ;
for my $f ( grep {m{(Tabs|Session)$}n} glob '*' ) {
say $f ;
copy( $f, $target ) or die qq{Copy Failed: $! } ;
}
</pre>
<br />
There's a bit of generalization in this. I would prefer to make it discern what TZ you're in without hard-coding it, and I <i>could</i> just use UTC and everyone would know to look for the most recent one.
<br />
<br />
The Lifehacker page points to a Windows-specific (or maybe Windows and Mac, I dunno) <code>Local State</code> file as where you set <i>exited_cleanly</i> to false in order to force recovery, while in Linux it's <code>Preferences</code>.
<br />
<br />
Where this leaves us is a bunch of files in the directory. I could figure out a way to parse the SNSS. I could create an HTML page showing that time's open tabs, which obviates the need to force Chrome to open the correct tabs. I could begin to start grouping them by window and time, saying "You've been wanting to get back to MQTT for some time; fish or cut bait".<br /><br />Plus, this example is very Linux-centric, meant to service Chromium and run in crontab. Making it run on Win7 or Win10 and Strawberry Perl and be run in Task Manager is important, as really, my Windows machine is for browsing and testing.<br />
<br />
If all I get is "Hey, give me my tabs back!", I'll be happy.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com1tag:blogger.com,1999:blog-145332831106508186.post-34566211083739415772017-05-22T15:51:00.000-04:002017-05-22T15:51:29.542-04:00Testing Perl Modules on Windows: A Cry For HelpI've been working on modules again, after a recent push, and I found a big project whose <code>.travis.yml</code> file only went to Perl 5.20. I thought I'd get some dev brownie points by adding the next two stable versions, and found that my build was dying in the Pull Request submission process.<br />
<br />
Specifically, it was dying on the checks. It passes <a href="http://travis-ci.org/" rel="nofollow" target="_blank">Travis-CI</a>, which runs the tests on Unix-like systems, but it was failing with Appveyor.<br />
<br />
"What is Appveyor?", I asked myself.<br />
<br />
Perhaps this isn't a direct quote.<br />
<br />
<a href="https://www.appveyor.com/" rel="nofollow" target="_blank">Appveyor</a> is a continuous integration service that tests against Windows systems. <a href="https://www.hanselman.com/" rel="nofollow" target="_blank">Scott Hanselman</a> wrote <a href="https://www.hanselman.com/blog/AppVeyorAGoodContinuousIntegrationSystemIsAJoyToBehold.aspx" rel="nofollow" target="_blank">a glowing review of it three years ago</a>.<br />
<br />
But there's no <code>.appveyor.yml</code> file in the project, so it runs and fails.<br />
<br />
I've mentioned this on the project IRC, and no, I'm not going to name names, because there is movement toward testing on Windows, and even if it doesn't work, I admire the goal.<br />
<br />
I wrote this three years ago, in response to a Python conference video:<br />
<blockquote class="tr_bq">
2) Sure, <i>real programmers</i> use Unix/Linux to run their code, <i>real programmers</i>, but beginner programmers don't come in knowing how to set up an environment and come in with the (Windows) computer they have, and the documentation sucks and they feel lost and they don't like it and they don't feel the power, and they're gone. Even if you dislike Microsoft, good tools for Windows and good documentation are important for new programmers, important for building community, important for inclusiveness.</blockquote>
I run Windows, but I program on and for Linux, and only have <a href="https://github.com/jacoby/SetBackgroundImage" rel="nofollow" target="_blank">one project based on Windows</a>. But I have installed ActiveState and Strawberry Perls, and think that if you write general-purpose modules, you should be sure to test against Windows as well as Linux.<br />
<br />
But, <a href="https://docs.travis-ci.com/user/languages/perl/" target="_blank">Travis-CI has documentation covering Perl projects</a>. <a href="https://www.appveyor.com/docs/build-environment/#perl" rel="nofollow" target="_blank">Appveyor says you can use Perl 5.20</a>. <a href="http://blogs.perl.org/users/eserte/" rel="nofollow" target="_blank">eserte</a> wrote <a href="http://blogs.perl.org/users/eserte/2016/04/testing-with-appveyor.html" target="_blank">a post on Appveyor for blogs.perl.org</a> last year, but I'd love to see better documentation from either them, the Perl community or both. Following is the YAML from eserte, with a switch to "check only master branch", but as with Travis, which uses perlbrew and allows testing as far back as 5.8.8, I think having it test against older versions of Perl, both ActivePerl and Strawberry, would be the thing.<br />
<br />
<pre class="yaml" name="code">branches:
only:
- master
skip_tags: true
cache:
- C:\strawberry
install:
- if not exist "C:\strawberry" cinst strawberryperl
- set PATH=C:\strawberry\perl\bin;C:\strawberry\perl\site\bin;C:\strawberry\c\bin;%PATH%
- cd C:\projects\%APPVEYOR_PROJECT_NAME%
- cpanm --installdeps .
build_script:
- perl Makefile.PL
- dmake test
</pre>
<br />
If you have a more fully-featured .appveyor.yml file you'd like the Perl community to use, especially distinguishing MakeMaker modules from Dist::Zilla modules, I'd love to see it.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-53666909420328378032017-05-17T09:35:00.005-04:002017-05-17T09:35:53.448-04:00Contact Me! (If you REALLY need to)A recent comment from Schlomi Fish said:
<br />
<blockquote>
Hi! I cannot seem to find any contact information on this page. How should I contact you?</blockquote>
And then linked to a <a href="http://www.shlomifish.org/meta/FAQ/#obscure_email_addr" rel="nofollow" target="_blank">FAQ entry</a> explaining his position on the state of email, comparing the futility of hiding addresses and the benefits of being open.<br />
<br />
I have to say, I hadn't thought about this in ... years? In general, I'm active on Twitter (<a href="https://twitter.com/jacobydave/" target="_blank">@jacobydave</a>), which is good if you are, but not helpful if you aren't. I try to keep track of the comments, but that doesn't fit every message a person would want to send me.<br />
<br />
So, a friendly "Hey, you should put your email on your blog" comment makes sense to me.<br />
<br />
But, adding more traffic to the mailbox that friends and relatives have access to doesn't. I'm happy to put up an email address, but I'm less than happy to make it my <i>main</i> email address. It goes to context; my coworkers generally don't get that one either.<br />
<br />
I had a long, barely touched by me but used enough by others project that was <a href="https://varlogrant.blogspot.com/2016/05/death-of-project.html" rel="nofollow" target="_blank">R syntax highlighting in Komodo Edit, which is dead</a> because it's now native, but the ActiveState packaging used an email address to set the id, so, I created rlangsyntax@gmail.com.<br />
<br />
So, to the right, in a section called "More Of Me", there is a requested <b>mailto:</b> pointing to <a href="mailto:rlangsyntax@gmail.com">rlangsyntax@gmail.com</a>. I will check it. Use it in good health.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-69271349017691425842017-05-08T12:09:00.003-04:002017-05-08T12:09:53.620-04:00Coffee and Code and Calendars and RI have what you might call <a href="https://varlogrant.blogspot.com/2012/06/my-caffeine-addiction.html" target="_blank">a conflicted relationship with caffeine</a>.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6-Zffo9rkUJjlxVBsi57iWAcsVGVmDEF2ryp75ZsPFYYjN8qTPUzgMlXQxAS30Kuno_-A4ejTSEsracQVzLxr9240DPNcOfWMziCWesVPQtb_RBZ1ad8KnqEtSg6B_0jp7ZDw3wXlUdke/s1600/2cups.jpg" imageanchor="1"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6-Zffo9rkUJjlxVBsi57iWAcsVGVmDEF2ryp75ZsPFYYjN8qTPUzgMlXQxAS30Kuno_-A4ejTSEsracQVzLxr9240DPNcOfWMziCWesVPQtb_RBZ1ad8KnqEtSg6B_0jp7ZDw3wXlUdke/s640/2cups.jpg" width="640" /></a>
<br />
<br />
Long story short: I found it necessary to cut down on caffeine, limiting myself to two cups a day, preferrably before noon, and, as sort of a measure of accountability, and as a way to gain more skill with SQL and R, I wrote tools that store when I drink coffee and, at the end of the work day, tweeting the image.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhv7GEQWHLM4vGcErXisqKXNSznkh0IcBBubpD5F7LQzgtBGU79-xiS1hiMocoQYbPuaFFeXVYqn5ZaOB_tptLws1d2r8NfB_rpLYVDO3MHQO3_ulCy2EopuvGF26Rfx8PeHpq0O7_QDoC1/s1600/coffee.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhv7GEQWHLM4vGcErXisqKXNSznkh0IcBBubpD5F7LQzgtBGU79-xiS1hiMocoQYbPuaFFeXVYqn5ZaOB_tptLws1d2r8NfB_rpLYVDO3MHQO3_ulCy2EopuvGF26Rfx8PeHpq0O7_QDoC1/s640/coffee.png" width="640" /></a></div>
<br />
I forget exactly where I found the calendar heatmap code, but it was several years ago, and was one of the first instances of ggplot2 that I found and put into service. I chose a brown-to-black color scheme because, well, obviously it needed to look like coffee.<br />
<br />
This image, with "Dave has had <i>n</i> cups of coffee today" is autotweeted every day at 6pm Eastern. Recently, it has drawn interest.<br />
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
<a href="https://twitter.com/hoelzro">@hoelzro</a> It's very cargo-cult on my end. I should blog or gist it. <br /><br />Thank <a href="https://twitter.com/hadleywickham">@hadleywickham</a> , I just put data into it.</div>
— 'Dev' in Creole (@JacobyDave) <a href="https://twitter.com/JacobyDave/status/860929736083722245">May 6, 2017</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script><br />
So here it is, both blogged and gisted.
<br />
<br />
<script src="https://gist.github.com/jacoby/ebfcfbc555776e544af1e671098adcfe.js"></script><br />
I started doing that thing with YAML in Perl, to keep database keys out of programs. I'm reasonably okay with my SQL skills, I think, but I am clear that my R code is largely cargo-cult. It'd be good to replace 2,4,6 with the days of the week, and I am reasonably sure that it's reversed from the way you'd normally expect weekdays to go. Some day, I'll have the knowledge required to make those changes. <br />
<br />
If you have questions and comments about this, I'd be glad to take them on, but I'm very much the learner when it comes to R.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-56451074169979571372017-05-08T11:28:00.002-04:002017-05-20T15:10:33.057-04:00count is not uniq...I recall reading something online about 20 years ago (gasp!) where the authors were looking for a core set of knowledge that would constitute "knowing Unix", and found that there just wasn't. Knowing Unix was like the Humpty Dance, in that no two people do it the same.<br />
<br />
And, presumably, you have Unix down when you appear to be in pain.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGuu636IgIZS2CBebliNfnXlNWLnGc-OnkHLB0OX6mFQokFzZ2JjSjsJbo0cebRNzXdFpVc7nPtcU-VlvJH2d2cjRsm1rCRgkU_tBGumwQU9z6K_KHZo5COLL6FqbLL3qypkuB7cGu9e0p/s1600/humpty+hump.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGuu636IgIZS2CBebliNfnXlNWLnGc-OnkHLB0OX6mFQokFzZ2JjSjsJbo0cebRNzXdFpVc7nPtcU-VlvJH2d2cjRsm1rCRgkU_tBGumwQU9z6K_KHZo5COLL6FqbLL3qypkuB7cGu9e0p/s400/humpty+hump.jpg" width="400" /></a></div>
<br />
I have been a Unix/Linux user since the 1990s and I only found out about <code>uniq -c</code> because of the last post. I had been using <code>sort | uniq</code> forever, and have recently stopped in favor of <code>sort -u</code>, which I also learned about recently.<br />
<br />
I find that <code>uniq</code> is kinda useless without a <code>sort</code> in front of it; if your input is "foo foo foo bar foo" (with requisite newlines, of course), <code>uniq</code> without <code>sort</code> will give you "foo bar foo" instead of "foo bar" or "bar foo", either of which is closer to what I want.<br />
<br />
So, I could see adding <code>alias count=" sort | uniq "</code> to my bash setup, but adding a count program to my <code>~/bin</code> seems much better to me, much closer to the Right Thing.<br />
<br />
marc chantreux suggested an implementation of <code>count</code> that is perhaps better and certainly shorter than the one I posted. There was regex magic that I simply didn't need, because I wanted the count to stand by itself (but I might revisit to remove the <code>awk</code> step, because as a user, I'm still a bit awkward with it.)<br />
<br />
B)<br />
<br />
<pre class="perl" name="code">my %seen ;
map { $seen{$_}++ } do {
@ARGV ? @ARGV : map { chomp ; $_ } <>;
} ;
while ( my ( $k, $v ) = each %seen ) {
say join "\t", $v, $k ;
}
</pre>
I like marc's use of the ternary operator to handle STDIN vs @ARGV, but I'm somewhat inconsistently against <code>map</code> over <code>for</code>. I know people who thing that <code>map</code> not going into an array is a problem, so I don't go back to it often. <br />
<br />
I do, however, do <code> for my $k ( keys %seen ) { ... } </code> enough that I'm sort of mad at myself for not encountering <code>each</code> before.
<br />
<br />
<b>ETA:</b> It's been brought to my attention that <a href="http://perl-begin.org/tutorials/bad-elements/#map_instead_of_foreach" rel="nofollow" target="_blank">using <code>map {}</code> as a replacement for <code>for () {}</code> is not good</a>.<br />
<br />
<br />Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com1tag:blogger.com,1999:blog-145332831106508186.post-91813858588868089832017-05-05T13:02:00.000-04:002017-05-05T13:02:33.611-04:00One! One New Utility! Bwa-ha-hahaha!Classic unix utilities give you a number of great tools, and you can use <code>sed</code> and <code>awk</code> and <code>bash</code> when those aren't enough.<br />
<br />
But sometimes ...<br />
<br />
I use <code>~</code> as a scratch space all too often, which leaves me with a huge amount of files that I stopped playing with a while ago. I can get to the point of knowing what types, sure, as I show here.<br />
<br />
<pre class="text" name="code">$ ls *.* | awk -F. '{ print $NF }'
jpg
jpg
jpg
jpg
txt
txt
txt
pl
txt
txt
pl
txt
pl
pl
txt
html
pl
pl
gz
mp3
pl
pl
pl
pl
txt
pl
pl
sh
sh
txt
pl
pl
diff
txt
txt
txt
pl
txt
pl
txt
txt
txt
txt
py
...
</pre>
<br />
But this only gets you so far. I can sort and know that there's a LOT of Perl files, perhaps too many, but nothing was immediate about telling me how many.<br />
<br />
But hey, I am a programmer, so I wrote a solution.<br />
<br />
<script src="https://gist.github.com/jacoby/95c4828171b45c259b73f9595cae6246.js"></script>
And here it is in a shell, combined with <code>sort</code> in order to give me the numbers, which includes a <i>lot</i> of throwaway Perl programs.<br />
<br />
<pre class="text" name="code">$ ls *.* | awk -F. '{ print $NF }' | count | sort -nr
95 pl
59 txt
10 sh
10 jpg
8 py
6 html
6 csv
5 js
2 gz
2 diff
1 zip
1 ttf
1 tt
1 svg
1 sql
1 Rd
1 R
1 pub
1 png
1 pdf
1 mp4
1 mp3
1 log
1 json
1 foo
1 conf
1 cnf
</pre>
<br />
<br />
I suppose I need to do some cleanup in <code>$HOME</code> today...Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-77200130835495895902017-03-14T19:07:00.001-04:002017-03-14T19:07:12.617-04:00Coding for Pi DayToday is Pi Day, which is a good day to talk about Pi.<br />
<br />
Normally, I'd probably use Pi, sine and cosine to draw things, but instead, I flashed on a couple ways to estimate Pi.<br />
<br />
Also, showing you can use Unicode characters in Perl.<br />
<br />
<pre class="perl" name="code">#!/usr/bin/env perl
use feature qw{ say } ;
use strict ;
use warnings ;
use utf8 ;
my $π = 3.14159 ;
my $est2 = estimate_2() ;
my $diff2 = sprintf '%.5f',abs $π - $est2 ;
say qq{Estimate 2: $est2 - off by $diff2} ;
my $est1 = estimate_1() ;
my $diff1 = sprintf '%.5f',abs $π - $est1 ;
say qq{Estimate 1: $est1 - off by $diff1} ;
exit ;
# concept here is that the area of a circle = π * rsquared
# if r == 1, area = π. If we just take the part of the circle
# where x and y are positive, that'll be π/4. So, take a random
# point between 0,0 and 1,1 see if the distance between it and
# 0,0 is < 1. If so, we increment, and the count / the number
# so far is an estimate of π.
# because randomness, this will change each time you run it
sub estimate_1 {
srand ;
my $inside = 0.0 ;
my $pi ;
for my $i ( 1 .. 1_000_000 ) {
my $x = rand ;
my $y = rand ;
$inside++ if $x * $x + $y * $y < 1.0 ;
$pi = sprintf '%.5f', 4 * $inside / $i ;
}
return $pi ;
}
# concept here is that π can be estimated by 4 ( 1 - 1/3 + 1/5 - 1/7 ...)
# so we get closer the further we go
sub estimate_2 {
my $pi = 0;
my $c = 0;
for my $i ( 0 .. 1_000_000 ) {
my $j = 2 * $i + 1 ;
if ( $i % 2 == 1 ) { $c -= 1 / $j ; }
else { $c += 1 / $j ; }
$pi = sprintf '%.5f', 4 * $c ;
}
return $pi ;
}
</pre>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-20111998988285844992017-02-28T18:41:00.000-05:002017-02-28T18:45:36.482-05:00Having Problems Munging Data in R<pre class="rstat" name="code">
#!/group/bioinfo/apps/apps/R-3.1.2/bin/Rscript
# a blog post in code-and-comment form
# Between having some problems with our VMs and wanting
# to learn Log::Log4perl. I wrote a program that took
# the load average -- at first at the hour, via
# crontab -- and stored the value. And, if the load
# average was > 20, it would send me an alert
# It used to be a problem. It is no longer. Now I
# just want to learn how to munge data in R
# read in file
logfile = read.table('~/.uptime.log')
# The logfile looks like this:
#
# 2017/01/01 00:02:01 genomics-test : 0.36 0.09 0.03
# 2017/01/01 00:02:02 genomics : 0.04 0.03 0.04
# 2017/01/01 00:02:02 genomics-db : 0.12 0.05 0.01
# 2017/01/01 00:02:04 genomics-apps : 1.87 1.24 0.79
# 2017/01/01 01:02:02 genomics-db : 0.24 0.14 0.05
# 2017/01/01 01:02:02 genomics-test : 0.53 0.14 0.04
# 2017/01/01 01:02:03 genomics : 0.13 0.09 0.08
# 2017/01/01 01:02:04 genomics-apps : 1.66 1.82 1.58
# 2017/01/01 02:02:01 genomics-test : 0.15 0.03 0.01
# ...
# set column names
colnames(logfile)=c('date','time','host','colon','load','x','y')
# now:
#
# date time host colon load x y
# 2017/01/01 00:02:01 genomics-test : 0.36 0.09 0.03
# 2017/01/01 00:02:02 genomics : 0.04 0.03 0.04
logfile$datetime <- paste( as.character(logfile$date) , as.character(logfile$time) )
# datetime == 'YYYY/MM/DD HH:MM:SS'
logfile$datetime <- sub('......$','',logfile$datetime)
# datetime == 'YYYY/MM/DD HH'
logfile$datetime <- sub('/','',logfile$datetime)
# datetime == 'YYYYMM/DD HH'
logfile$datetime <- sub('/','',logfile$datetime)
# datetime == 'YYYYMMDD HH'
logfile$datetime <- sub(' ','',logfile$datetime)
# datetime == 'YYYYMMDDHH'
# for every datetime in logfile. I love clean data
# removes several columns we no longer need
logfile$time <- NULL
logfile$date <- NULL
logfile$colon <- NULL
logfile$x <- NULL
logfile$y <- NULL
# logfile now looks like this:
#
# datetime host load
# 2017010100 genomics-test 0.36
# 2017010100 genomics 0.04
# 2017010100 genomics-db 0.12
# 2017010100 genomics-apps 1.87
# 2017010101 genomics-db 0.24
# 2017010101 genomics-test 0.53
# 2017010101 genomics 0.13
# 2017010101 genomics-apps 1.66
# 2017010102 genomics-test 0.15
# ...
# and we can get the X and Y for a big huge replacement table
hosts <- unique(logfile$host[order(logfile$host)])
dates <- unique(logfile$datetime)
# because what we want is something closer to this
#
# datetime genomics genomics-apps genomics-db genomics-test
# 2017010100 0.04 1.87 0.12 0.36
# 2017010101 0.13 1.66 0.15 0.53
# ...
# let's try to put it into a dataframe
uptime.data <- data.frame()
uptime.data$datetime <- vector() ;
for ( h in hosts ) {
uptime.data[h] <- vector()
}
# and here, we have a data frame that looks like
#
# datetime genomics genomics-apps genomics-db genomics-test
#
# as I understand it, you can only append to a data frame by merging.
# I need to create a data frame that looks like
#
# datetime genomics genomics-apps genomics-db genomics-test
# 2017010100 0.04 1.87 0.12 0.36
#
# and then merge that. Then do the same with
#
# datetime genomics genomics-apps genomics-db genomics-test
# 2017010101 0.13 1.66 0.15 0.53
#
# and so on.
#
# I don't know how to do that.
#
# I *think* the way is make a one-vector data frame:
#
# datetime
# 2017010101
#
# and add the vectors one at a time.
for ( d in dates ) {
# we don't and the whole log here. we just want
# this hour's data
#
# datetime host load
# 2017010100 genomics-test 0.36
# 2017010100 genomics 0.04
# 2017010100 genomics-db 0.12
# 2017010100 genomics-apps 1.87
log <- subset(logfile, datetime==d)
print(d)
for ( h in hosts ) {
# and we can narrow it down further
#
# datetime host load
# 2017010100 genomics 0.04
hostv <- subset(log,host==h)
load = hostv$load
# problem is, due to fun LDAP issues, sometimes
# the logging doesn't happen
if ( 0 == length(load) ) { load <- -1 }
print(paste(h, load ))
}
# and here's where I'm hung. I can get all the pieces
# I want, even -1 for missing values, but I can't seem
# to put it together into a one-row data frame
# to append to uptime.data.
# [1] "2017010100"
# [1] "genomics 0.04"
# [1] "genomics-apps 1.87"
# [1] "genomics-db 0.12"
# [1] "genomics-test 0.36"
# [1] "2017010101"
# [1] "genomics 0.13"
# [1] "genomics-apps 1.66"
# [1] "genomics-db 0.24"
# [1] "genomics-test 0.53"
# [1] "2017010102"
# [1] "genomics 0.36"
# [1] "genomics-apps 0.71"
# [1] "genomics-db 0.08"
# [1] "genomics-test 0.15"
}
</pre>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-44577927581923943802017-01-20T15:38:00.003-05:002017-01-20T15:38:57.017-05:00Ding! Ding! The Process is Dead!Starts with a thing I saw on <a href="https://davidwalsh.name/ding" target="_blank">David Walsh's Blog</a>:
<br />
<blockquote>
I've been working with beefy virtual machines, docker containers, and build processes lately. Believe it or not, working on projects aimed at making Mozilla developers more productive can mean executing code that can take anywhere from a minute to an hour, which in itself can hit how productive I can be. For the longer tasks, I often get away from my desk, make a cup of coffee, and check in to see how the rest of the Walsh clan is doing.
<br />
<br />
When I walk away, however, it would be nice to know when the task is done, so I can jet back to my desk and get back to work. My awesome Mozilla colleague Byron "glob" Jones recently showed me his script for task completion notification and I forced him to put it up on GitHub so you all can get it too; it's called <a href="https://github.com/globau/ding/" target="_blank">ding!</a></blockquote>
OK, that sounds cool. So I go to Github and I see one line that gives me pause.<br />
<br />
<code>Requires ding.mp3 and error.mp3 in same directory as script. OSX only.</code>
<br />
<br />
I can handle the image thing, but I don't own or run an OSX computer. (I have one somewhere, but it's ancient and has no functioning battery. I don't use it.)<br />
<br />
"So," I think, "how could I do this on my Linux box? What's the shortest path toward functionality on this concept?"<br />
<br />
Well, recently, I have been playing with Text-to-Speech. Actually, I have been a long-time user of TTS, using <code>festival</code> then <code>espeak</code> to tell me the current time and temperature on the hour and half-hour. I switched to Amazon's <a href="https://aws.amazon.com/polly/">Polly</a> in December, deciding that the service sounded much better than the on-my-computer choices. (<a href="http://web.ics.purdue.edu/~djacoby/tts.html">Hear for yourself.</a>) So, I knew how to handle the audio aspects.<br />
<br />
The other part required me to get much more familiar with Perl's <code>system</code> function than I had been previously.
<br />
<br />
<script src="https://gist.github.com/jacoby/4cbeb63606bd47e6df138f95b7958cab.js"></script>
<br />
I'm not yet 100% happy with this code, but I'm reasonably okay with it so far. Certainly the concept has been proven. (I use the audio files from globau's ding.) With enough interest, I will switch it from being a GitHub gist to being a repo.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-36739185733236987312016-11-19T14:32:00.001-05:002016-11-19T14:32:18.322-05:00Graphs are not that Scary!<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4jD_wBh9BMkaEsjzi2ekZQNhS2vXDZadfmz68yt7xFTAGv6LjBPNgKYGuuKt5baAB7KzXwDueKXrNa0k5mUQBy5TRkgK3fcnMvlVZhFo8iKwMbu7kB-HrYkAcnxJzOG0Ms8bYFD4MseEg/s1600/2016-11-19+14.25.07.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4jD_wBh9BMkaEsjzi2ekZQNhS2vXDZadfmz68yt7xFTAGv6LjBPNgKYGuuKt5baAB7KzXwDueKXrNa0k5mUQBy5TRkgK3fcnMvlVZhFo8iKwMbu7kB-HrYkAcnxJzOG0Ms8bYFD4MseEg/s640/2016-11-19+14.25.07.jpg" width="640" /></a></div>
<br /><br />As with most things I blog about, this starts with Twitter. I follow a lot of people on Twitter, and I use Lists. I want to be able to group people more-or-less on community, because there's the community where they talk about programming, for example, and the community where they talk about music, or the town I live in.<br />
<br />
I can begin to break things up myself, but curation is a hard thing, so I wanted to do it automatically. And I spent a long time not knowing what to do. I imagined myself traversing trees in what looks like linked lists reimagined by Cthulhu, and that doesn't sound like much fun at all.<br />
<br />
Eventually, I decided to search on "graphs and Perl". Of course, I probably should've done it earlier, but oh well. I found <a href="https://metacpan.org/pod/Graph" target="_blank">Graph</a>. I had used <a href="https://metacpan.org/pod/GD::Graph" target="_blank">GD::Graph</a> before, which is a plotting library. (There has to be some index of how overloaded words are.) And once I installed it, I figured it out: <b>As a programmer, all you're dealing with are arrays and hashes. Nothing scary.</b><br />
<br />
<h3>
<b>Word Ladder</b></h3>
<br />
We'll take a problem invented by Lewis Carroll called a "word ladder", where you find your way from one word (for example, "cold") to another ("warm") by changing one letter at a time:<br />
<br />
<b> cold</b><br />
<b> coRd</b><br />
<b> cArd</b><br />
<b> Ward</b><br />
<b> warM</b><br />
<br />
Clearly, this can and is often done by hand, but if you're looking to automate it, there are three basic problems: what are the available words, how do you determine when words are one change away, and how do you do this to get the provable shortest path?<br />
<br />
First, I went to <a href="ftp://ftp.cerias.purdue.edu/pub/dict/" target="_blank">CERIAS</a> years ago and downloaded word lists. Computer security researchers use them because real words are bad passwords, so, lists of real words can be used to create rainbow tables and the like. My lists are years old, so there may be new words I don't account for, but unlike Lewis Carroll, I can get from APE to MAN in five words, not six.<br />
<br />
<b> ape</b><br />
<b> apS</b><br />
<b> aAs</b><br />
<b> Mas</b><br />
<b> maN</b><br />
<br />
Not sure that Lewis Carroll would've accepted AAS, but there you go<br />
<br />
There is a term for the number of changes it takes to go from one word to another, and it's called the <a href="https://en.wikipedia.org/wiki/Levenshtein_distance" target="_blank">Levenshtein Distance</a>. I first learned about this from perlbrew, which is how, if you type "perlbrew isntall", it guesses that you meant to type "perlbrew install". It's hardcoded there because perlbrew can't assume you have anything but perl and core modules. I use the function from perlbrew instead of <a href="https://metacpan.org/pod/Text::Levenshtein" target="_blank">Text::Levenshtein</a> but it is a module worth looking into.<br />
<br />
And the final answer is <i>"Put it into a graph and use <a href="https://en.wikipedia.org/wiki/Dijkstra's_algorithm" target="_blank">Dijkstra's Algorithm!</a>"</i><br />
<br />
Perhaps not with the exclamation point.<br />
<br />
<h3>
Showing Code</h3>
<br />
Here's making a graph of it:<br />
<br />
<pre class="perl" name="code">#!/usr/bin/env perl
use feature qw{say} ;
use strict ;
use warnings ;
use Data::Dumper ;
use Graph ;
use List::Util qw{min} ;
use Storable ;
for my $l ( 3 .. 16 ) {
create_word_graph($l) ;
}
exit ;
# -------------------------------------------------------------------
# we're creating a word graph of all words that are of length $length
# where the nodes are all words and the edges are unweighted, because
# they're all weighted 1. No connection between "foo" and "bar" because
# the distance is "3".
sub create_word_graph {
my $length = shift ;
my %dict = get_words($length) ;
my @dict = sort keys %dict ; # sorting probably is unnecessary
my $g = Graph->new() ;
# compare each word to each word. If the distance is 1, put it
# into the graph. This implementation is O(N**2) but probably
# could be redone as O(NlogN), but I didn't care to.
for my $i ( @dict ) {
for my $j ( @dict ) {
my $dist = editdist( $i, $j ) ;
if ( $dist == 1 ) {
$g->add_edge( $i, $j ) ;
}
}
}
# Because I'm using Storable to store the Graph object for use
# later, I only use this once. But, I found there's an endian
# issue if you try to open Linux-generated Storable files in
# Strawberry Perl.
store $g , "/home/jacoby/.word_$length.store" ;
}
# -------------------------------------------------------------------
# this is where we get the words and only get words of the correct
# length. I have a number of dictionary files, and I put them in
# a hash to de-duplicate them.
sub get_words {
my $length = shift ;
my %output ;
for my $d ( glob( '/home/jacoby/bin/Toys/Dict/*' ) ) {
if ( open my $fh, '<', $d ) {
for my $l ( <$fh> ) {
chomp $l ;
$l =~ s/\s//g ;
next if length $l != $length ;
next if $l =~ /\W/ ;
next if $l =~ /\d/ ;
$output{ uc $l }++ ;
}
}
}
return %output ;
}
# -------------------------------------------------------------------
# straight copy of Wikipedia's "Levenshtein Distance", straight taken
# from perlbrew. If I didn't have this, I'd probably use
# Text::Levenshtein.
sub editdist {
my ( $f, $g ) = @_ ;
my @a = split //, $f ;
my @b = split //, $g ;
# There is an extra row and column in the matrix. This is the
# distance from the empty string to a substring of the target.
my @d ;
$d[ $_ ][ 0 ] = $_ for ( 0 .. @a ) ;
$d[ 0 ][ $_ ] = $_ for ( 0 .. @b ) ;
for my $i ( 1 .. @a ) {
for my $j ( 1 .. @b ) {
$d[ $i ][ $j ] = (
$a[ $i - 1 ] eq $b[ $j - 1 ]
? $d[ $i - 1 ][ $j - 1 ]
: 1 + min( $d[ $i - 1 ][ $j ], $d[ $i ][ $j - 1 ], $d[ $i - 1 ][ $j - 1 ] )
) ;
}
}
return $d[ @a ][ @b ] ;
}
</pre>
<br />
Following are what my wordlists can do. Something tells me that, when we get to 16-letter words, it's more a bunch of disconnected nodes than a graph.<br />
<br />
<b>1718 3-letter words</b><br />
<b>6404 4-letter words</b><br />
<b>13409 5-letter words</b><br />
<b>20490 6-letter words</b><br />
<b>24483 7-letter words</b><br />
<b>24295 8-letter words</b><br />
<b>19594 9-letter words</b><br />
<b>13781 10-letter words</b><br />
<b>8792 11-letter words</b><br />
<b>5622 12-letter words</b><br />
<b>3349 13-letter words</b><br />
<b>1851 14-letter words</b><br />
<b>999 15-letter words</b><br />
<b>514 16-letter words</b><br />
<br />
My solver isn't perfect, and the first thing I'd want to add is ensuring that both the starting and ending words are actually in the word list. Without that, your code goes on forever.<br />
<br />
So, I won't show off the whole program below, but it does use <a href="https://metacpan.org/pod/Storable" target="_blank">Storable</a>, Graph and feature qw{say}.<br />
<br />
<pre class="perl" name="code">dijkstra( $graph , 'foo' , 'bar' ) ;
# -------------------------------------------------------------------
# context-specific perl implementation of Dijkstra's Algorithm for
# shortest-path
sub dijkstra {
my ( $graph, $source, $target, ) = @_ ;
# the graph pre-exists and is passed in
# $source is 'foo', the word we're starting from
# $target is 'bar', the word we're trying to get to
my @q ; # will be the list of all words
my %dist ; # distance from source. $dist{$source} will be zero
my %prev ; # this holds our work being every edge of the tree
# we're pulling from the graph.
# we set the the distance for every node to basically infinite, then
# for the starting point to zero
for my $v ( $graph->unique_vertices ) {
$dist{$v} = 1_000_000_000 ; # per Wikipeia, infinity
push @q, $v ;
}
$dist{$source} = 0 ;
LOOP: while (@q) {
# resort, putting words with short distances first
# first pass being $source , LONG WAY AWAY
@q = sort { $dist{$a} <=> $dist{$b} } @q ;
my $u = shift @q ;
# say STDERR join "\t", $u, $dist{$u} ;
# here, we end the first time we see the target.
# we COULD get a list of every path that's the shortest length,
# but that's not what we're doing here
last LOOP if $u eq $target ;
# this is a complex and unreadable way of ensuring that
# we're only getting edges that contain $u, which is the
# word we're working on right now
for my $e (
grep {
my @a = @$_ ;
grep {/^${u}$/} @a
} $graph->unique_edges
) {
# $v is the word on the other end of the edge
# $w is the distance, which is 1 because of the problem
# $alt is the new distance between $source and $v,
# replacing the absurdly high number set before
my ($v) = grep { $_ ne $u } @$e ;
my $w = 1 ;
my $alt = $dist{$u} + $w ;
if ( $alt < $dist{$v} ) {
$dist{$v} = $alt ;
$prev{$v} = $u ;
}
}
}
my @nodes = $graph->unique_vertices ;
my @edges = $graph->unique_edges ;
return {
distances => \%dist,
previous => \%prev,
nodes => \@nodes,
edges => \@edges,
} ;
}</pre>
<br />
I return lots of stuff, but the part really necessary is %prev, because that, $source and $target are everything you need. Assuming we're trying to go from <b>FOR </b>to <b>FAR</b>, a number of words will satisfy <code>$prev{FOR}</code>, but it's the one we're wanting. In the expanded case of <b>FOO</b> to <b>BAR</b>, <code>$prev->{BAR}</code> = 'FAR', <code>$prev->{FAR}</code> is 'FOR', and <code>$prev->{FOR}</code> is 'FOO'.<br />
<br />
And nothing in there is complex. It's all really hashes or arrays or values. Nothing a programmer should have any problem with.<br />
<br />
CPAN has a number of other modules of use: <a href="https://metacpan.org/pod/Graph::Dijkstra" target="_blank">Graph::Dijkstra</a> has that algorithm already written, and <a href="https://metacpan.org/pod/Graph::D3" target="_blank">Graph::D3</a> allows you to create a graph in such a way that you can use it in <a href="http://d3.js/">D3.js</a>. Plus, there are a number of modules in Algorithm::* that do good and useful things. So go in, start playing with it. It's deep, there are weeds, but it isn't scary.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-19313957022645212752016-11-08T10:54:00.000-05:002016-11-08T10:54:08.031-05:00Modern Perl but not Modern::PerlThis started while driving to work. If I get mail from coworkers, I get <a href="https://metacpan.org/pod/Net::Pushover" target="_blank">Pushover</a> notifications, and halfway from home, I got a bunch of notifications.
<br />
<div>
<br /></div>
<div>
We don't know the cause of the issue, but I do know the result</div>
<div>
<br /></div>
<div>
We have env set on our web server set so that Perl is <code>/our/own/private/bin/perl</code> and not <code>/usr/bin/perl</code>, because this runs in a highly-networked and highly-clustered environment, mostly RedHat 6.8 and with 5.10.1 as system perl, if we want to have a consistent version and consistent modules, we need our own. This allows us to have <code>#!/usr/bin/env perl</code> as our shbang.<br />
<br />
And this morning, for reasons I don't know, it stopped working. Whatever perl was being called, it wasn't <code>/our/own/private/bin/perl</code>. And this broke things.<br />
<br />
One of the things that broke is this: Whatever perl is <code>/usr/bin/env perl</code>, it doesn't have <code><a href="https://metacpan.org/pod/Modern::Perl" target="_blank">Modern::Perl</a></code>. <br />
<br />
I'm for Modern Perl. My personal take is that chromatic and <a href="http://modernperlbooks.com/" target="_blank">Modern Perl</a> kept Perl alive in with Perl 5 while Larry Wall and the other language developers worked on Perl 6. Thus, I am grateful that it exists. But, while I was playing with it, I found a problem: <code>Modern::Perl</code> is not in <a href="http://perldoc.perl.org/index-modules-M.html" target="_blank">Core</a>, so you cannot rely on it being there, so a script might be running with a version greater than 5.8.8 and be able to give you everything you need, which to me is normally <code>use strict</code>, <code>use warnings</code> and <code>use feature qw{say}</code>, but if you're asking for Modern::Perl for it, it fails, and because you don't know which Modern thing you want, you don't know how to fix it.<br />
<br />
This is part of my persistent hatred of magic. If it works and you don't understand how, you can't fix it if it stops working. I got to the heavy magic parts of Ruby and Rails and that, as well as "Life Happens", are why I stopped playing with it. And, I think, this is a contributing factor with this morning's brokenness.</div>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-31897586401754393582016-10-30T14:59:00.001-04:002016-10-30T15:00:44.738-04:00Net::Twitter Cookbook: Favorites and FollowersFavorites.<br />
<br />
Also known as "Likes", they're an indication in Twitter that you approve of a status update. Most of the time, they're paired with retweets as signs by the audience to the author that the post is agreeable. Like digital applause.<br />
<br />
This is all well and good, but it could be used for so much more, if you had more access and control over them.<br />
<br />
So I did.<br />
<br />
The first step is to collect them. There's an API to get them, and collecting them in bulk is easy. A problem is avoiding grabbing the same tweet twice.<br />
<br />
<pre class="perl" name="code"># as before, the "boilerplate" can be found elsewhere in my blog.
use IO::Interactive qw{ interactive } ;
my $config ;
$config->{start} = 0 ;
$config->{end} = 200 ;
for ( my $page = $config->{start}; $page <= $config->{end}; ++$page ) {
say {interactive} qq{\tPAGE $page} ;
my $r = $twit->favorites( {
page => $page ,
count => 200 ,
} ) ;
last unless @$r ;
# push @favs , @$r ;
for my $fav (@$r) {
if ( $config->{verbose} ) {
say {interactive} handle_date( $fav->{created_at} )
}
store_tweet( $config->{user}, $fav ) ;
}
sleep 60 * 3 ; # five minutes
}
</pre>
<br />
Once I had a list of my tweets, one of the first things I did was use them to do "Follow Friday". If you know who you favorited over the last week, it's an easy thing to get a list of the usernames, count them and add them until you have reached the end of the list or 140 characters.<br />
<br />
Then, as I started playing with APIs and wanted to write my own ones, I created an API to find ones containing a substring, like <i>json</i> or <i>pizza</i> or <i>sleep</i>. This way, I could begin to use a "favorite" as a bookmark.<br />
<br />
(I won't show demo code, because I'm not happy or proud of the the code, which lives in a trailing-edge environment, and because it's more database-related than Twitter-focused.)<br />
<br />
As an aside, I do not follow back. There are people who follow me who I have no interest in reading, and there are people I follow who care nothing about my output. In general, I treat Twitter as something between a large IRC client and an RSS reader, and I never expected nor wanted RSS feeds to track me.<br />
<br />
But this can be a thing worth tracking, which you can do, without any storage, with the help of a list. Start with getting a list of those following you, those you follow, and the list of accounts (I almost wrote "people", but that isn't guaranteed) in your follows-me list. If they follow you and aren't in your list, add them. If they're in the list and you have started following them, take them out. If they're on the list and aren't following you, drop them. As long as you're not big-time (Twitter limits lists to 500 accounts), that should be enough to keep a Twitter list of accounts you're not following.<br />
<br />
<pre class="perl" name="code">use List::Compare ;
my $list = 'id num of your Twitter list';
my $followers = $twit->followers_ids() ;
my @followers = @{ $followers->{ids} } ;
my $friends = $twit->friends_ids() ;
my @friends = @{ $friends->{ids} } ;
my @list = get_list_members( $twit, $list ) ;
my %list = map { $_ => 1 } @list ;
my $lc1 = List::Compare->new( \@friends, \@followers ) ;
my $lc2 = List::Compare->new( \@friends, \@list ) ;
my $lc3 = List::Compare->new( \@followers, \@list ) ;
# if follows me and I don't follow, put in the list
say {interactive} 'FOLLOWING ME' ;
for my $id ( $lc1->get_complement ) {
next if $list{$id} ;
add_to_list( $twit, $list, $id ) ;
}
# if I follow, take off the list
say {interactive} 'I FOLLOW' ;
for my $id ( $lc2->get_intersection ) {
drop_from_list( $twit, $list, $id ) ;
}
# if no longer following me, take off the list
say {interactive} 'NOT FOLLOWING' ;
for my $id ( $lc3->get_complement ) {
drop_from_list( $twit, $list, $id ) ;
}
#========= ========= ========= ========= ========= ========= =========
sub add_to_list {
my ( $twit, $list, $id ) = @_ ;
say STDERR qq{ADDING $id} ;
eval { $twit->add_list_member(
{ list_id => $list, user_id => $id, } ) ; } ;
if ($@) {
warn $@->error ;
}
}
#========= ========= ========= ========= ========= ========= =========
sub drop_from_list {
my ( $twit, $list, $id ) = @_ ;
say STDERR qq{REMOVING $id} ;
eval {
$twit->delete_list_member( { list_id => $list, user_id => $id, } ) ;
} ;
if ($@) {
warn $@->error ;
}
}
</pre>
<br />
But are there any you should follow? Are there any posts in the the feed that you might "like"? What do you "like" anyway?<br />
<br />
There's a way for us to get an idea of what you would like, which is your past likes. First, we must get, for comparison, a collection of what your Twitter feed is like normally. (I grab 200 posts an hour and store them. This looks and works exactly like my "grab favorites code", except I don't loop it.<br />
<br />
<pre class="perl" name="code"> my $timeline = $twit->home_timeline( { count => 200 } ) ;
for my $tweet (@$timeline) {
my $id = $tweet->{id} ; # twitter_id
my $text = $tweet->{text} ; # text
my $created = handle_date( $tweet->{created_at} ) ; # created
my $screen_name = $tweet->{user}->{screen_name} ; # user id
if ( $config->{verbose} ) {
say {interactive} handle_date( $tweet->{created_at} );
say {interactive} $text ;
say {interactive} $created ;
say {interactive} $screen_name ;
say {interactive} '' ;
}
store_tweet( $config->{user}, $tweet ) ;
# exit ;
}
</pre>
<br />
So, we have a body of tweets that you like, and a body of tweets that are a representative sample of what Twitter looks like to you. On to <a href="https://metacpan.org/pod/Algorithm::NaiveBayes" target="_blank">Algorithm::NaiveBayes</a>!<br />
<br />
<pre class="perl" name="code">use Algorithm::NaiveBayes ;
use IO::Interactive qw{ interactive } ;
use String::Tokenizer ;
my $list = 'ID of your list';
my $nb = train() ;
my @top = read_list( $config, $nb , $list ) ;
say join ' ' , (scalar @top ), 'tweets' ;
for my $tweet (
sort { $a->{analysis}->{fave} <=> $b->{analysis}->{fave} } @top ) {
my $fav = int $tweet->{analysis}->{fave} * 100 ;
say $tweet->{text} ;
say $tweet->{user}->{screen_name} ;
say $tweet->{gen_url} ;
say $fav ;
say '' ;
}
exit ;
#========= ========= ========= ========= ========= ========= =========
# gets the first page of your Twitter timeline.
#
# avoids checking a tweet if it's 1) from you (you like yourself;
# we get it) and 2) if it doesn't give enough tokens to make a
# prediction.
sub read_list {
my $config = shift ;
my $nb = shift ;
my $list = shift ;
...
my @favorites ;
my $timeline = $twit->list_statuses({list_id => $list});
for my $tweet (@$timeline) {
my $id = $tweet->{id} ; # twitter_id
my $text = $tweet->{text} ; # text
my $created = handle_date( $tweet->{created_at} ) ; # created
my $screen_name = $tweet->{user}->{screen_name} ; # user id
my $check = toke( lc $text ) ;
next if lc $screen_name eq lc $config->{user} ;
next if !scalar keys %{ $check->{attributes} } ;
my $r = $nb->predict( attributes => $check->{attributes} ) ;
my $fav = int $r->{fave} * 100 ;
next if $fav < $config->{limit} ;
my $url = join '/', 'http:', '', 'twitter.com', $screen_name,
'status', $id ;
$tweet->{analysis} = $r ;
$tweet->{gen_url} = $url ;
push @favorites, $tweet ;
}
return @favorites ;
}
#========= ========= ========= ========= ========= ========= =========
sub train {
my $nb = Algorithm::NaiveBayes->new( purge => 1 ) ;
my $path = '/home/jacoby/.nb_twitter' ;
# adapted on suggestion from Ken to
# gets all tweets in your baseline table
my $baseline = get_all() ;
for my $entry (@$baseline) {
my ( $tweet, $month, $year ) = (@$entry) ;
my $label = join '', $year, ( sprintf '%02d', $month ) ;
my $ham = toke(lc $tweet) ;
next unless scalar keys %$ham ;
$nb->add_instance(
attributes => $ham->{attributes},
label => ['base'],
) ;
}
gets all tweets in your favorites table
my $favorites = get_favorites() ;
for my $entry (@$favorites) {
my ( $tweet, $month, $year ) = (@$entry) ;
my $label = join '', $year, ( sprintf '%02d', $month ) ;
my $ham = toke(lc $tweet) ;
next unless scalar keys %$ham ;
$nb->add_instance(
attributes => $ham->{attributes},
label => ['fave'],
) ;
}
$nb->train() ;
return $nb ;
}
#========= ========= ========= ========= ========= ========= =========
# tokenizes a tweet by breaking it into characters, removing URLs
# and short words
sub toke {
my $tweet = shift ;
my $ham ;
my $tokenizer = String::Tokenizer->new() ;
$tweet =~ s{https?://\S+}{}g ;
$tokenizer->tokenize($tweet) ;
for my $t ( $tokenizer->getTokens() ) {
$t =~ s{\W}{}g ;
next if length $t < 4 ;
next if $t !~ /\D/ ;
my @x = $tweet =~ m{($t)}gmix ;
$ham->{attributes}{$t} = scalar @x ;
}
return $ham ;
}
</pre>
<br />
Honestly, <a href="https://metacpan.org/pod/String::Tokenizer" target="_blank">String::Tokenizer</a> is probably a bit too overkill for this, but I'll go with it for now. It might be better to get a list of the 100 or 500 most common words and exclude them from the tweets, instead of limiting by size. As is, strings like <i>ada</i> and <i>sql</i> would be excluded. But it's good for now.<br />
<br />
We get a list of tweets including a number between 0 and 1, representing the likelihood, by Bayes, that I would <i>like</i> the tweet. In the end, it's turned into an integer between 0 and 100. You can also run this against your normal timeline to pull out tweets you would've liked but missed. I often do this<br />
<br />
I run the follows_me version on occasion. So far, it is clear to me that the people I don't follow, I don't follow for a reason, and that remains valid.<br />
<br />
If you use this and find value in it, please tell me below. Thanks and good coding.Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-23426554259820340342016-10-28T14:45:00.001-04:002016-10-28T14:45:28.683-04:00Using the Symbol Table: "Help"?I've been looking at command-line code for both fun and work. I know I can have one module handle just the interface, and have the module where the functionality happens pass the functionality along.<br />
<br />
<pre class="perl" name="code">
#!/usr/bin/env perl
use feature qw'say state' ;
use strict ;
use warnings ;
use utf8 ;
my $w = Wit->new( @ARGV ) ;
$w->run() ;
package Wit ;
use lib '/home/jacoby/lib' ;
use base 'Witter' ;
use Witter::Twitter ;
1;
</pre>
<br />
<pre class="perl" name="code">
package Witter ;
# highly adapted from perlbrew.
use feature qw{ say } ;
use strict ;
use warnings ;
sub new {
my ( $class, @argv ) = @_ ;
my $self ;
$self->{foo} = 'bar' ;
$self->{args} = [] ;
if (@argv) {
$self->{args} = \@argv ;
}
return bless $self, $class ;
}
sub run {
my ($self) = @_ ;
$self->run_command( $self->{args} ) ;
}
sub run_command {
my ( $self, $args ) = @_ ;
if ( scalar @$args == 0
|| lc $args->[0] eq 'help'
|| $self->{help} ) {
$self->help(@$args) ;
exit ;
}
if ( lc $args->[0] eq 'commands' ) {
say join "\n\t", '', $self->commands() ;
exit ;
}
my $command = $args->[0] ;
my $s = $self->can("twitter_$command") ;
unless ($s) {
$command =~ y/-/_/ ;
$s = $self->can("twitter_$command") ;
}
unless ($s) {
my @commands = $self->find_similar_commands($command) ;
if ( @commands > 1 ) {
@commands = map { ' ' . $_ } @commands ;
die
"Unknown command: `$command`. Did you mean one of the following?\n"
. join( "\n", @commands )
. "\n" ;
}
elsif ( @commands == 1 ) {
die "Unknown command: `$command`. Did you mean `$commands[0]`?\n"
;
}
else {
die "Unknown command: `$command`. Typo?\n" ;
}
}
unless ( 'CODE' eq ref $s ) { say 'Not a valid command' ; exit ; }
$self->$s(@$args) ;
}
sub help {
my ($self,$me,@args) = @_ ;
say 'HELP!' ;
say join "\t", @args;
}
sub commands {
my ($self) = @_ ;
my @commands ;
my $package = ref $self ? ref $self : $self ;
my $symtable = do {
no strict 'refs' ;
\%{ $package . '::' } ;
} ;
foreach my $sym ( sort keys %$symtable ) {
if ( $sym =~ /^twitter_/ ) {
my $glob = $symtable->{$sym} ;
if ( defined *$glob{CODE} ) {
$sym =~ s/^twitter_// ;
$sym =~ s/_/-/g ;
push @commands, $sym ;
}
}
}
return @commands ;
}
# Some functions removed for sake of brevity
</pre>
<br />
<pre class="perl" name="code">
package Witter::Twitter ;
use strict ;
use feature qw{ say state } ;
use warnings FATAL => 'all' ;
use Exporter qw{import} ;
use Net::Twitter ;
use JSON::XS ;
our $VERSION = 0.1 ;
our @EXPORT ;
for my $entry ( keys %Witter::Twitter:: ) {
next if $entry !~ /^twitter_/mxs ;
push @EXPORT, $entry ;
}
sub twitter_foo {
my ( $self, @args ) = @_ ;
say "foo" ;
say join '|', @args ;
}
1 ;
</pre>
<br />
And the above works when called as below.
<br />
<pre class="perl" name="code">
jacoby@oz 13:49 60°F 51.24,-112.49 ~
$ ./witter
HELP!
jacoby@oz 13:52 60°F 51.25,-94.51 ~
$ ./witter help
HELP!
jacoby@oz 13:53 60°F 50.59,-88.64 ~
$ ./witter commands
foo
jacoby@oz 13:53 60°F 50.59,-88.64 ~
$ ./witter help foo
HELP!
foo
jacoby@oz 13:53 60°F 50.59,-88.64 ~
$ ./witter foo
foo
foo
jacoby@oz 13:53 60°F 50.59,-88.64 ~
$ ./witter moo
Unknown command: `moo`. Did you mean `foo`?
</pre>
<br />
In the above example, I'm just doing the one add-on module, <code>Witter::Twitter</code> and one function, <code>Witter::Twitter::foo</code>, but clearly, I would want it open-ended, so that if someone wanted to add <code>Witter::Facebook</code>, all the information about the Facebook functions would be in that module.
<br>
<br>
Then, of course, I would have to use another prefix than <code>twitter_</code>, but we'll leave that, and ensuring that modules don't step on each others' names, to another day.
<br>
<br>
The part that concerns me is <code>help</code>. Especially <code>help foo</code>. It should be part of the the module it's in; If <code>Witter::Twitter</code> is the module with <code>foo()</code>, only it should be expected to know about <code>foo()</code>.
<br>
<br>
But how to communicate it? I'm flashing on <code>our %docs</code> and <code>$docs{foo}= 'This is foo, silly'</code> but the point of the whole thing is to allow the addition of modules that the initial module doesn't know about, and it would require knowing to look for <code>%Witter::Twitter::docs</code>.
<br>
<br>
I suppose adding a <code>docs_</code> function that looks like this.
<br>
<pre class="perl" name="code">
sub docs_foo {
return q{
This explains the use of the 'foo' command
...
};
END
}
</pre>
<br>
<br>
I'm diving into this in part because I have code that uses basically this code, and I need to add functionality to it, and while I'm in there, I might as well make user documentation better. Or even possible.
<br>
<br>
I'm also parallel-inspired by looking at a Perl project built on and using old Perl ("require 5.005") and recent blog posts about Linus Torvalds and "Good Taste". There's something tasteful about being able to add <code>use Your::Module</code> and nothing else to code, but if the best I can do is knowledge that there's a <code>foo</code> command, with no sense of what it does, that seems like the kind of thing that Linus would rightfully curse me for.
<br>
<br>
Is there a better way of doing things like this? I recall there being interesting things in <code>Net::Tumblr</code> that would require me to step up and learn <code>Moose</code> or the like. This is yet another important step toward me becoming a better and more tasteful programmer, but not germane to today's ranting.
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0tag:blogger.com,1999:blog-145332831106508186.post-13152274913606787702016-10-15T13:27:00.000-04:002016-10-15T13:27:12.727-04:00Gender and Wearables?<blockquote class="twitter-tweet" lang="en">
<div dir="ltr" lang="en">
<a href="https://twitter.com/hashtag/wearables?src=hash">#wearables</a> etc (arduino, others) is the rare tech where women are present and often set the pace of the tech. hardware more welcoming? ;)</div>
— Su-Shee (@sheeshee) <a href="https://twitter.com/sheeshee/status/618340512278425600">July 7, 2015</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<br />
<div>
First I heard about modern wearables was at Indiana Linuxfest, where the speaker went on about the coming wave of microcontrollers and posited a belt buckle that was aware of when it was pointing toward magnetic north and activate a haptic feedback device, so that, for the wearer, eventually true sense of direction would eventually become another sense.<br />
<br />
I'm sure I could find a sensor that could tell me that, that could ship from China and cost me less than a gumball. I'm sure I could easily get a buzzer, that I could control it all with a small <a href="http://arduino.cc/" target="_blank">Arduino</a> board like a <a href="http://www.adafruit.com/categories/261" target="_blank">Trinket</a> or <a href="http://www.adafruit.com/categories/92" target="_blank">Flora</a> or <a href="https://www.arduino.cc/en/Main/ArduinoBoardNano" target="_blank">Nano</a> or the like. and <a href="http://www.jimmydiresta.com/" target="_blank">Jimmy DiResta</a> has already taught me <a href="https://www.youtube.com/watch?v=M3wge7upWDM" target="_blank">how to make a belt buckle</a>. And I actually kinda want one. But I haven't made it.<br />
<br />
In part it's because my available resources to push toward projects like this are small at the moment. In part, though, it's because, once I put on my watch, my tablet, my glasses and <a href="http://varlogrant.blogspot.com/2009/01/carrying-things.html" target="_blank">the Leatherman on my belt</a>, I'm accessorized out.<br />
<br />
I think most American men are about the same. </div>
Dave Jacobhttp://www.blogger.com/profile/15052163927020492687noreply@blogger.com0