Cookie Notice

As far as I know, and as far as I remember, nothing in this page does anything with Cookies.

2016/08/31

Overkill III: Permutations of Overkill (Perl6)


Long-time readers remember my previous code with the Magic Box. In short, given the numbers [ 3, 4, 5, 6, 7, 8, 9, 10, 11 ] , arrange them in a 3x3 square such that every row, column and diagonal adds up to 21.

My first pass, on a very old computer, had the C version taking a second and a half, the Perl5 version taking a minute and a half, and the Perl6 version taking an hour and a half. But I no longer use that computer, and am now using a much newer one, so I took the opportunity to try again.

I have Nvidia cards in my box, so I could eventually write Overkill IV about that, but I don't have the knowledge or time right now. I did install rakudobrew and the newest version of Perl6, but otherwise, everything was unchanged.

C dropped to 0.6 seconds. Perl5 now takes 16 seconds. Perl6 now takes 10 minutes. So, the newer computer brought substantial speed gains for each, but the most dramatic is with Perl6, which tells us that the Perl6 developers have not only worked on correctness and completeness, but speed. Good for them! Good for us!

But wait! There's more!

@loltimo saw the post and suggested that I use permutations.

"What are permutations?", you ask? That's what basically happens in recursive_magic_box(), where all the possible variations are created. Given A, B and C, there are six possible orderings: ABC ACB BAC BCA CAB CBA. If I was given all possible orderings of an array, rather than (mis)handling it on my own and having to check that I got it right, would that make things faster?

As it turns out, yes. 47 seconds, rather than 10 minutes. More than 10x faster. But still, not as fast as Perl5. I now want to implement permutations for Perl5 and see if it makes that one faster.

And, it makes me feel much better about Perl6.

Code Samples Below

2016/08/29

Perl and cpanm on Bash on Ubuntu on Windows 10: Who do I report this to?

I promised a longer post on Bash on Ubuntu on Windows 10, explaining why I use it, when I don't and why I even use Windows 10, but that isn't today.

Today is me trying to figure out who to report a bug to, and how to effectively report it.

The system (called "Ubuntu on Windows" for the rest of this blog post) is Ubuntu 14.04 LTS running kinda with Windows as the kernel instead of Linux. A good thing is that, if you can apt-get install a package, you can put it on Ubuntu on Windows. I don't think you can use it if 1) it connects to the kernel (because the kernel is Windows) or 2) it pops up a window. I've heard about people using XMing to get around that, but I haven't tried it yet.

The problem is that you get what you have available with dpkg, and the perl that comes with 14.04 is 5.18, where current is 5.24. Some modules haven't been packaged, and there's usually a gap between the package version and the cpan version, so I tend to use perlbrew and use my own perl instead of system perl.

And cpan and cpanm can't build on Ubuntu on Windows. I demonstrated this by trying to install YAML::XS with cpanm. I won't include the build log, but I will link to the build log. The problem comes at lines 146, 162 and 207:
#   Failed test 'Literal Scalar'
#   at t/dump-heuristics.t line 4.
#          got: '--- '1234567890
# 
#   1234567890
# 
#   1234567890
# 
# '
# '
#     expected: '--- |
#   1234567890
#   1234567890
#   1234567890
# '

#   Failed test 'Double Quoted Scalar'
#   at t/dump-heuristics.t line 10.
#          got: '--- 'A
# 
#   B
# 
#   C
# 
# '
# '
#     expected: '--- "A\nB\nC\n"
# '
# Looks like you failed 2 tests of 2.
t/dump-heuristics.t ...... 

...

#   Failed test 'Dumping Chinese hash works'
#   at t/utf8.t line 31.
#          got: '---
# Email: boss@opcafe.net
# 地址: 新竹市 300 石坊街 37-8 號
# 店名: OpCafé
# 時間: 11:01~23:59
# 電話: '03-5277806
# 
#   0991-100087
# 
# '
# '
#     expected: '---
# Email: boss@opcafe.net
# 地址: 新竹市 300 石坊街 37-8 號
# 店名: OpCafé
# 時間: 11:01~23:59
# 電話: "03-5277806\n0991-100087\n"
# '
# Looks like you failed 1 test of 8.
t/utf8.t ................. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/8 subtests 

The problem involves incorrectly handling newlines, but all this works and YAML::XS installs perfectly on my personal Perl, which I demonstrate below.
$ which perl ; perl -v | head -n 5 ; perl -MYAML::XS -e "print 'Yet Another Perl Hacker'"

/home/jacoby/perl5/perlbrew/perls/perl-5.24.0/bin/perl

This is perl 5, version 24, subversion 0 (v5.24.0) built for i686-linux
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2016, Larry Wall
Yet Another Perl Hacker

So, if it's Perl on Ubuntu on Windows, and the Perl works perfectly other places (including Strawberry Perl on the same Windows 10 machine (but only in PowerShell) ) and the Ubuntu works perfectly on other places, it must be the Windows, right?

That's what I thought, too. There's a GitHub repo for Ubuntu on Windows. Well, really, it's an issue tracker for Ubuntu on Windows, because there's nothing else in there. You can't clone and build your own. But, it does put the issue tracker in a place where the natural audience for Ubuntu on Windows would go. I put in an issue, but I think I might have asked wrong. I have cleaned up my issue some, but I see no sign that anyone else has seen it.

So, what is my next step? It isn't just YAML::XS; that's just a module I wanted at the time. It isn't just cpanm; I tried with cpan, too. It is just with Windows 10. Thoughts and suggestions from the greater Perl community?

2016/08/19

On Not YET Contributing to an Open Source Project

Not the module in question. 

I had an idea, and the shortest path between here and a working implementation is through CPAN, so I found a module and tried to install it.

No go. Failed tests.

So I find the GitHub repo and make an issue.

That doesn't get me any closer to a working implementation of my idea, nor does it involve me coding or running code. We can't have that, so I took the next step, which was forking, cloning and branching the code, then playing with it to find out what's going on and why.

I won't tell you which repo; that's really besides the point. I've dealt with Perl enough to know that, when the module was last updated, everything was tested and everything worked. There is therefore no condemnation for those who are in CPAN.

Simply speaking, at some point, the API the module is interacting with changed how it works, returning an image instead of JSON that would contain the URL of said image. Not an unreasonable thing to do, I think. That saves you a step, and for my purposes, I'd never have need to get either the image or the location of the image. I feel free to believe that force installing the module would get me
an acceptable outcome.

But if I know a thing is broken and I know how to fix it, and I don't fix it, I'm simply leaving the world in a broken state. That's hardly a responsible response.

The question becomes "Just how do I fix this?", and I see three choices:
  • FOLLOW THE TEST! The test wants an object which is converted from the JSON the API returns. I could easily skip the API call and just return http://example.com/{whatever}/image/. This will even be a valid URL, but when the API changes again, which it will, the URL will be inaccurate. This seems brittle to me.
  • CHANGE THE TEST! The API wants to pass back an image. I can pass that on through and rewrite the test. But the module isn't brand new, which means that someone out there is using the failing function to get the URL of the image and this change will brake that existing code. It's probably already broken; this module won't install, as established, but this is a significant change in the API, which shouldn't be made without consideration. 
  • DROP IT ALL! Remove the function that causes the problem. Remove the tests. Leave only a stub saying "This functionality has been removed". In some ways, this is the coward's way out, but it would be the simplest thing. Easier to remove functionality than to add it.
After I thought about it, I was leaning toward an All-Three approach: Making three branches, implementing each idea, then submitting three parallel pull requests, leaving it to the maintainer to decide the proper choice.

Yeah, that's not a smart plan. 2/3 of that work is going to be not used, by definition. After consultation with my advisors (read, anyone who would listen on Twitter, IRC or Hangouts), I decided that asking before coding anything, so I added details to my (admittedly short and incomplete, submitted before understanding) issue to say "I see three alternatives; what do you want me to do?"

But that is the proper way to do it. It isn't like I went "I want a feature, and here it is." I just want a working module to do the thing I'm thinking of (which might not be as cool as I thought it was), and the things I could do to make it work again could bug the people actually using it. The community of users.

every change breaks someone's workflow
As always, this is a point where xkcd understands and explains all.
So, I have been thinking about this, asking about this and writing about this, rather than implementing the initial idea, or even fixing the problem.

Which frustrates me to no end.

2016/08/12

A Little Fun with OpenCV


I have a webcam that I have set to take pictures at work. Is this a long-term art project? Is it a classic Twitter over-share? An example of Quantified Self gone horribly, horribly wrong? I honestly don't know.

I do know that it's brought me into interesting places before. Getting my camera to work with V4L. Tweeting images. Determining if the screen is locked, both on Unity and Gnome. I've come to the point where I can set @hourly ~/bin/tweet_a_picture.pl. I don't, rather 0 9-18 * * 1-5 ~/bin/tweet_a_picture.pl, because if it's before 9am or on a weekend, I won't be here, so don't even try.

But if I'm pulled away, there are photos like the above. Who needs to see my space without me?

Yeah, who needs to see my space with me? We'll pass on that.

One of the members of Purdue Perl Mongers formerly worked with OpenCV, and I asked him to present on that topic this month. He even created a GitHub repo with his example code. It's Python, not Perl, which will be a digression for this blog, but not an unprecedented one. This gave me enough confidence in OpenCV and how it worked to create a program that uses it to tell if anyone's in the frame.

(The demo code suggested installing python-opencv for Ubuntu, to which I must add that the crucial opencv-data does not come with it. Just a warning.)

# webcam_here.py

# usage: webcam_here.py
# 1

# when runs, returns the number of upper bodies, and thus people, it currently sees

import cv2
# pass either device id (0 usually for webcam) or path to a video file
cam = cv2.VideoCapture(0)

# face cascade bundled with OpenCV
# classifier = cv2.CascadeClassifier('/usr/share/opencv/haarcascades/haarcascade_frontalface_default.xml')
classifier = cv2.CascadeClassifier('/usr/share/opencv/haarcascades/haarcascade_upperbody.xml')
# I found that haarcascade_upperbody did the best for identifying me.

# read in frame
retval, frame = cam.read()

# convert to grayscale
frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

# run classifier
results = classifier.detectMultiScale(
 frame_gray,
 # let's assume face will be close/big if in front of webcam
 minSize=(300,300)
)

# number of found upper bodies in webcam
print len(results)


And here it is in context.
# tweet_a_picture.pl
use 5.010 ;
use strict ;
use warnings ;
use IO::Interactive qw{ interactive } ;
use Net::Twitter ;
use YAML::XS qw{ DumpFile LoadFile } ;
use Getopt::Long ;

use lib '/home/jacoby/lib' ;
use Locked qw{ is_locked there };

my $override = 0 ;

GetOptions(
    'override' => \$override ,
    ) ;

my $config_file = $ENV{ HOME } . '/.twitter.cnf' ;
my $config      = LoadFile( $config_file ) ;

my $user = 'screen_name' ;
my $status = 'Just took a picture' ;

if ( is_locked() == 0 && there() == 1 || $override ) {
    say { interactive } 'unlocked' ;
    # highest supported resolution of my old-school webcam 
    # Twitter DOES have a 5MB limit on file size
    # https://dev.twitter.com/rest/media/uploading-media#imagerecs
    my $cam = q{/usr/bin/fswebcam -q -r 640x480 --jpeg 85 -D 1 --no-banner} ;
    my $time = time ;
    my $image = "/home/jacoby/Pictures/Self/shot-${time}.jpeg" ;
    qx{$cam $image} ;
    tweet_pic( $status , $image ) ;
    say { interactive } $time ;
    }

sub tweet_pic {
    my ( $status , $pic ) = @_ ;
    my ( $access_token, $access_token_secret ) ;
    ( $access_token, $access_token_secret ) = restore_tokens( $user ) ;
    my $twit = Net::Twitter->new(
        traits              => [qw/API::RESTv1_1/],
        consumer_key    => $config->{ consumer_key },
        consumer_secret => $config->{ consumer_secret },
        ssl => 1 ,
        ) ;

    if ( $access_token && $access_token_secret ) {
        $twit->access_token( $access_token ) ;
        $twit->access_token_secret( $access_token_secret ) ;
        }
    unless ( $twit->authorized ) {
        say qq{UNAUTHTORIZED} ; exit ;
        # I really handle this better
        # http://varlogrant.blogspot.com/2016/07/nettwitter-cookbook-how-to-tweet.html
        }

    my $img_ref ;
    my $file = $pic ;
    my ( $filename ) = reverse split m{/} , $pic ;
    push @$img_ref , $file ;
    push @$img_ref , $filename ;

    if ( $twit->update_with_media( $status , $img_ref  ) ) {
        no warnings ;
        say { interactive } $status ;
        }
    else {
        say { interactive } 'FAIL' ;
        }
    }

# is in Locked.pm, but placed here for simplicity
# there are issues with recognizing my webcam, which 
# we quash by sending STDERR to /dev/null
sub there {
    my $checker = '/home/jacoby/bin/webcam_here.py' ;
    return -1 if ! -f $checker ;
    my $o = qx{$checker 2> /dev/null} ;
    chomp $o ;
    return $o ;
    }

sub restore_tokens {
    my ( $user ) = @_ ;
    my ( $access_token, $access_token_secret ) ;
    if ( $config->{ tokens }{ $user } ) {
        $access_token = $config->{ tokens }{ $user }{ access_token } ;
        $access_token_secret =
            $config->{ tokens }{ $user }{ access_token_secret } ;
        }
    return $access_token, $access_token_secret ;
    }

sub save_tokens {
    my ( $user, $access_token, $access_token_secret ) = @_ ;
    $config->{ tokens }{ $user }{ access_token }        = $access_token ;
    $config->{ tokens }{ $user }{ access_token_secret } = $access_token_secret ;
    DumpFile( $config_file, $config ) ;
    return 1 ;
    }


Clearly, this is the start, it's currently very cargo-cult code, and it has several uses. A coming use is to go through my archive of pictures and identifying the ones where I'm gone. That'll either require me getting up to speed with Image::ObjectDetect or Python, but it's a thing I'm willing to do.

An interesting side-issue: I've found that the existing Haar cascades (the XML files that define a thing OpenCV can identify) do not like my glasses, and thus cannot identify my eyes or face with them on, thus, me using the upperbody cascade. I think I should train my own Haar classifier; I know I have enough pics of me for it.

2016/08/06

Net::Twitter Cookbook: Images and Profiles

I've covered how you handle keys and authenticating with Twitter previously, so look into those for more information as we go ahead with sending tweets!

There was a time when Twitter was just text, so you could just send a link to an image. There were sites like TwitPix that hosted images for you, but eventually Twitter developed the ability to host media.

    my $media ;
    push @$media, $config->{ file } ;
    push @$media, 'icon.' . $suffix ;
    if ( $twit->update_with_media( $status , $media ) ) {
        say 'OK' ;
        }

There are four ways you could define the media:

    $media = [ 'path to media' ] ;
    $media = [ 'path to media', 'replacment filename' ] ;
    $media = [ 'path to media', 'replacment filename', 
            Content-Type => mime/type' ] ;
    $media = [  undef , 'replacment filename', 
            Content-Type => mime/type', 'raw media data'] ;


I use the second in order to specify the file name that isn't whatever long and random filename it has. The module has ways to guess the correct mime type, but you can specify to avoid that. The last one, starting with undef, allows you to create or modify an image with Image::Magick or whatever you choose and tweet it without having to involve the filesystem at all.

A few words about what we mean by media. We mean images or video; no audio file option. By images, we mean PNG, JPG, GIF or WEBP. No SVG, and less that 5MB as of this writing. (Check https://dev.twitter.com/ for more info later.)

Video has other constraints, but I don't tweet just video often, if ever. I generally put it to YouTube and post the link.

There's a couple ways you can have more fun with images on Twitter. Those would be with your profile image, the square image that sits next to your tweet, and the profile banner, the landscape image that shows up at the top of your page.

They work like the above, handling media the same way.

    my $image ;
    push @$image, $config->{ file } ;
    push @$image, 'icon.' . $suffix ;
    if ( $twit->update_profile_image( $image ) ) {
        say 'OK' ;
        }

    my $banner ;
    push @$banner, $config->{ file } ;
    push @$banner, 'icon.' . $suffix ;
    if ( $twit->update_profile_banner( $banner ) ) {
        say 'OK' ;
        }


Profile images are square, and are converted to square if your initial image is not square. For both, JPG, GIF or PNG are allowed and are converted upon upload. If you try to upload an animated GIF, they will use the first frame. This can make your profile image less fun, but if you have a feed full of pictures that throb and spin, that can make you feel seasick.

And, since we're hitting profile-related issues, perhaps I should handle your profiles and we can get this behind us.

Changing your username, going from @jacobydave to something else, is doable but not via the API. It isn't a thing you can or should be doing with an API. You can change other details, however. You can change your name ("Dave Jacoby"), description ("Guy blogging about Perl's Net::Twitter library. Thought Leader. Foodie."), website ("http://varlogrant.blogspot.com/") and location ("Dark Side of the Moon").

    use Getopt::Long ;

    my $config ;
    GetOptions(
        'description=s' => \$config->{ description },
        'location=s'    => \$config->{ location },
        'name=s'        => \$config->{ name },
        'web=s'         => \$config->{ url },
        ) ;

    my $params ; for my $p ( qw{ name url location description } ) {
    $params->{ $p } = $config->{ $p } if $config->{ $p } ; }
    $params->{ include_entities } = 0 ;  $params->{ skip_status } = 1
    ;

    if ( $twit->update_profile( $params ) ) {
        say 'OK' ;
        }
    }


I *DO* have fun fun with location, setting it with a number of strings; "My Happy Place" on Saturday, "Never got the hang..." for Thursdays. You can set this to any string, and Twitter does nearly nothing with it. I'd suggest you ignore it or set it to something general or clever, and forget it.

Now that we've covered how you tweet and handle your identity a little, next time we'll start to get into relationships, which are the things that make a social network social.

2016/08/02

Net::Twitter Cookbook: How I tweet, plus

Previously, I wrote a post showing the basics on how to send a tweet using Perl and Net::Twitter. I showed the easiest you can do it.

Below is the code that I use most often. It is a command-line tool, where it's used something along the lines of named_twitter.pl jacobydave text of my tweet. Except, that's not how I type it, thanks to my .alias file. I have twitter aliased to '~/bin/named_twitter jacobydave ' and normally tweet like twitter this is how I tweet.

This isn't to say I never automate tweets; I certainly do. But it is a rare part of what I do with the Twitter API and Net::Twitter. I will dive deeper into issues with tweeting and direct messages, both in a technical and social matter, in a later post.

But I said that you have the consumer key and secret, which identify you as a service, and the access token and secret, which identify you as a user of the service. In the above code sample, I use YAML to store this data, but you could use JSON, Sqlite or anything else to store it. Among other benefits, you can put your code into GitHub, as I did above, without exposing anything.

As I proceed, I will assume that tokens are handled somehow and proceed directly to the cool part.

Again, you get the consumer key by going to apps.twitter.com and creating a new app.


You log into apps.twitter.com with your Twitter account, click "Create New App" and fill in the details. When I created my three, you had to specifically choose "can send DMs" specifically, so if you're creating apps to follow along, I do suggest you allow yourself that option.