Cookie Notice

As far as I know, and as far as I remember, nothing in this page does anything with Cookies.
Showing posts sorted by date for query twitter. Sort by relevance Show all posts
Showing posts sorted by date for query twitter. Sort by relevance Show all posts

2017/07/25

Sentimentalizing Twitter: First Pass

It started here.

I couldn't leave it there.

I can write a tool that goes through all old tweets and deletes ones that don't pass criteria, but I prefer to get out ahead of the issue and leave the record as it is.

And, as a later pass, one can pull a corpus, use a module like Algorithm::NaiveBayes and make your own classifier, rather than using Microsoft Research's Text Analytics API or another service. I was somewhere along that process when the hosting died, so I'm not bringing it here.

I was kinda under the thrall of Building Maintainable Software, or at least the first few chapters of it, so I did a few things differently in order to get their functions less than 15 lines, but the send_tweet function didn't get the passes it'd need, and I could probably give check_status some love, or perhaps even roll it into something like WebService::Microsoft::TextAnalytics. In the mean time, this should allow you to always tweet on the bright side of life.

#!/usr/bin/env perl

use feature qw{ postderef say signatures } ;
use strict ;
use warnings ;
use utf8 ;

no warnings qw{ experimental::postderef experimental::signatures } ;

use Carp ;
use Getopt::Long ;
use IO::Interactive qw{ interactive } ;
use JSON ;
use LWP::UserAgent ;
use Net::Twitter ;
use YAML qw{ LoadFile } ;

my $options = options() ;
my $config  = config() ;

if ( check_status( $options->{ status }, $config ) >= 0.5 ) {
    send_tweet( $options, $config ) ;
    }
else { say qq{Blocked due to negative vibes, man.} }
exit ;

sub send_tweet ( $options, $config ) {
    my $twit = Net::Twitter->new(
        traits          => [ qw/API::RESTv1_1/ ],
        consumer_key    => $config->{ twitter }{ consumer_key },
        consumer_secret => $config->{ twitter }{ consumer_secret },
        ssl             => 1,
        ) ;
    my $tokens = $config->{ twitter }{ tokens }{ $options->{ username } } ;
    if (   $tokens->{ access_token }
        && $tokens->{ access_token_secret } ) {
        $twit->access_token( $tokens->{ access_token } ) ;
        $twit->access_token_secret( $tokens->{ access_token_secret } ) ;
        }
    if ( $twit->authorized ) {
        if ( $twit->update( $options->{ status } ) ) {
            say { interactive } $options->{ status } ;
            }
        else {
            say { interactive } 'FAILED TO TWEET' ;
            }
        }
    else {
        croak( "Not Authorized" ) ;
        }
    }

sub check_status ( $status, $config ) {
    my $j   = JSON->new->canonical->pretty ;
    my $key = $config->{ microsoft }{ text_analytics }{ key } ;
    my $id  = 'tweet_' . time ;
    my $object ;
    push @{ $object->{ documents } },
        {
        language => 'EN',
        text     => $status,
        id       => $id,
        } ;
    my $json    = $j->encode( $object ) ;
    my $api     = 'https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment' ;
    my $agent   = LWP::UserAgent->new ;
    my $request = HTTP::Request->new( POST => $api ) ;
    $request->header( 'Ocp-Apim-Subscription-Key' => $key ) ;
    $request->content( $json ) ;
    my $response = $agent->request( $request ) ;

    if ( $response->is_success ) {
        my $out = decode_json $response->content ;
        my $doc = $out->{ documents }[ 0 ] ;
        return $doc->{ score } ;
        }
    else {
        croak( $response->status_line ) ;
        }
    return 1 ;
    }

sub config () {
    my $config ;
    $config->{ twitter }   = LoadFile( join '/', $ENV{ HOME }, '.twitter.cnf' ) ;
    $config->{ microsoft } = LoadFile( join '/', $ENV{ HOME }, '.microsoft.yml' ) ;
    return $config ;
    }

sub options () {
    my $options ;
    GetOptions(
        'help'       => \$options->{ help },
        'username=s' => \$options->{ username },
        'status=s'   => \$options->{ status },
        ) ;
    show_usage( $options ) ;
    return $options ;
    }

sub show_usage ($options) {
    if (   $options->{ help }
        || !$options->{ username }
        || !$options->{ status } ) {
        say { interactive } <<'HELP';
Only Positive Tweets -- Does text analysis of content before tweeting

    -u  user        Twitter screen name (required)
    -s  status      Status to be tweeted (required)
    -h  help        This screen
HELP
        exit ;
        }
    }
__DATA__

.microsoft.yml looks like this
---
text_analytics:
    key: GENERATED_BY_MICROSOFT

.twitter.cnf looks like this

---
consumer_key: GO_TO_DEV.TWITTER.COM
consumer_secret: GO_TO_DEV.TWITTER.COM
tokens:
    your_username:
        access_token: TIED_TO_YOU_AS_USER_NOT_DEV
        access_token_secret:TIED_TO_YOU_AS_USER_NOT_DEV

I cover access_tokens in https://varlogrant.blogspot.com/2016/07/nettwitter-cookbook-how-to-tweet.html


2017/07/06

Working Through Limitations: The Perl Conference 2017 Day 3

A Delorean with a Perl-powered center column and the cutest little Flux Capacitor on the dashboard. Oh, the wonders you can see at a developer conference.
"A man's got to know his limitations."

That's a line from Dirty Harry Callahan in Magnum Force, but it really described my planning for the Perl Conference. Once the calendar was up, I went in, first and foremost thinking "What are skills I need to learn?"

One crucial skill is version control. It's difficult to add to my main workflow, as I develop in production. (I live in fear.) But I'm increasingly adding it to my side projects. It is especially part of the process for maintaining the site for Purdue Perl Mongers, as well as aspects of HackLafayette, but beyond certain basics, I just didn't know much about how to use Git and version control to improve my projects. I learned how CPAN Testers tests your code on many platforms after you upload to CPAN, and how Travis-CL and Appveyor test against Linux, macOS and Windows after pushing to GitHub, but how track changes, align issues with branches, etc., are all new to me. So, I started Wednesday with Genehack and Logs Are Magic: Why Git Workflows and Commit Structure Should Matter To You. (Slides) I fully expect to crib ideas and aliases from this talk for some time to come.



There was a talk at a local software group featuring a project leader from Amazon on the technology involved with Alexa, which involved a lot of how this works, going from speech-to-text to tokenization and identifying of crucial keywords -- "Alexa, have Domino's order me a Pizza" ultimately boiling down to "Domino's" and "Pizza" -- and proceeding from there. It gave a sense of how Amazon is taking several hard problems and turning them into consumer tools.

What came very late in the talk is how to interface my systems with Amazon's "Talking Donkey", and I had a few conversations where we talked about starting the day with "Alexa, what's fresh hell is this?" and getting back a list of systems that broke overnight, but I lacked a strong idea of what is needed to make my systems interact with the Alexa system.

And an Echo.

But, thankfully, Jason Terry's Amazon, Alexa and Perl talk covered this, albeit more in the "Turn my smart lights on" sense than in the "Tell me what my work systems are doing" sense. Still, very much something I had been interested in.



But, as I implied, Amazon does a lot of heavy lifting with Alexa, getting it down to GETs and and POSTs against your API. If you're running this from home, where you have a consistent wired connection, this works. But, if you're running this, for example, in your car, you need it to be small, easy, and self-contained. Chris Prather decided to MAKE New Friends with a Raspberry Pi 3. This was a case of Conference-Driven Development, and he didn't have it ready to demonstrate at this time.



I've been trying to move my lab to the New Hotness over time, and because I will have to tie things back to existing systems, I have avoided learning too much in order to make it work. Joel Berger presented Vue.js, Mojolicious, and PostgreSQL chat in less than 50 lines. I've heard good things about Vue.js, which I heard from the project head was the parts he needed from Angular without the stuff he didn't use. (RFC Podcast) I use jQuery and Template and not much more, so the "only what you need" aspect sounds good to me. I fully expect to bug Joel on Twitter and IRC about this and other things I need to do with Mojolicious  over the coming months. (Slides)



But, of course, not every talk needs to speak directly to my short-term needs. Stevan Little has been trying to add an object system to the Perl 5 for quite some time, and in Hold My Beer and Watch This, he talks about Moxie, his latest attempt.



OK, the Moxie talk was in the same room as the Mojolicious talk and this next one. I could have gone to one of the others, but I decided against. Oh well. I would put "do more object-oriented development" as a thing to learn, so I was glad to hear it.

The last full talk was The Variable Crimes We Commit Against JavaScript by Julka Grodel. I would say that I code Javascript half as much as I code Perl, but twice as much as I code in other languages. I knew certain parts, like how let gives lexical scoping like Perl's my. I had heard about let from my JS Twitter list, as well as from Matt Trout's talk, but there were certainly things I didn't know.

And, actually, things I still don't. I had technical difficulties with my laptop, and if I could've worked it out that day, I would've tried to set up a lightning talk. Alas, not only did it not work -- in the end, I swapped out the WiFi, and I think if I switched back, it'd be sane again -- I missed a lot of her talk about arrow functions, which piqued the interest of Perl's functional programming guru, Mark Jason Dominus. (In a not-surprising addition to my limitations, I still haven't gotten to the end of Higher Order Perl.) Anyway, I believe that, after I finish this post, I will go back and watch this again.



After this, there were Lightning Talks, interspersed with ads for various Perl Mongers groups, and urges for you to start a Perl Mongers group. I am likely going to do another post, with these talks from all three days, but to end this one, I'll show off one where M. Allan Noah talks about the SANE Project and how he reverse engineers scanner output to allow him to support scanners without vendor input.



Lack of documentation is a limitation, but knowing the limitation does not mean you have to stop, and lack of documentation will not stop him. We should all draw inspiration from that.

Now that I'm two weeks past the end, most of the talks are up, and I would like to commend the organizers. Conference organization is hard, and this venue made it harder. The US Patent and Trademark Office is a great place, but there are aspects that they as a venue wanted secured and I would've preferred to be more open, but it was a beautiful venue and I be glad to return.

But, the thing I'd like to commend them most on is the high quality and fast turnaround on talk videos. The audio is good, the video is clear and well-lit, and the slides take precedence over the speaker in framing. It's everything I want in a conference video.

2017/06/19

Feeling "Tipsy": The Perl Conference 2017 Day 1

A few lessons learned before I dive into the talks I attended and what I gained from them:

  • I had figured that aspects of my Every Day Carry would not be appreciated in a government building, so I left my multitool off my belt, but, even after I had put everything I could think of into the tray, I had still made the machines go beep, causing the guards to pull out the wand. Tomorrow, pack lighter.
  • I take notes on a paper notebook, but I used my phone and tablet to connect to others via IRC and Twitter, as well as keeping track of my calendar, and I ran through my batteries before the end of the talks. In part, it was switching between the local WiFi and the cellular network, doing well with neither, but I'm not convinced it'd wouldn't be drained regardless. At lunch, I need to find a place to charge. I often come with a power strip for just this purpose, too.
  • I didn't break out the laptop once. If I don't use it more, I should just leave it and have less to carry.
Hopefully, I will remember this and come prepared for the next one.

Now, to the talks:

I started the day with MetaCPAN, the Grand Tour. I had been convinced earlier that MetaCPAN is better than CPAN for looking into modules, but there's more, including modules I am going to have start using to search CPAN.  

Graham Knop's Continuous Integration for CPAN followed. I had been aware of Travis-CI, and had become aware of AppVeyor recently, but the tooling available in Perl to work with these was less familiar to me. I was unaware that you can specify Linux or OSX in Travis, This was something I was thinking and asking questions of other developers about. I have issues on FreeBSD, which I'm told is something that Gitlab-CI can help me with, but somehow, I doubt I can connect Github to Gitlab, but I could be wrong.

Steven Lembark had much more with Dockerizing CPAN Testers: Running an Isolated Test Site than I could fit into my head, and I think I'll have to go back to the tape once it's available, but I think it's a useful addition to the world.

After lunch, I went to Joel Berger's Variables, Scoping and Namespaces, which he set as a talk for beginners. He went so far as to suggest more established developers go elsewhere. Since I never thought I learned all of Perl when I was learning, it was very much a lot of things I did already know, but a little "So that's why we do that", some more "Oooh. I forgot about that", and one weird trick that explains how to mock functions for tests.

(That, fundamentally, is my big item to work on as a developer. Testing.)

After this, I attended Matt S. Trout's ES6: Almost an acceptable Perl 5? and it gave me a sense of treating Javascript like Perl, but since I don't code Perl like Matt does, I probably won't code ES6 like Matt does. My notes peter out about halfway through, but they do include several gems, such as lodash, that might improve the code I do write.

Following this is Lightning Talks, which has a bunch of interesting points, going from "Write for OpenSource.com" to "Try learning Dart" to "Creating Test JSON from JSON Schemas" to "Using Instapaper, IFTTT and Buffer to tweet what you read" to the Alien modules, which I almost understand now. Or maybe not. Certainly, though, I'd be installing and trying Dave Rolsky's RTx-toGitHub right now if I wasn't so tired.

Finally, Sawyer X talked about Perl 5.26 and the changes that came and the changes that are coming. The thing that comes to mind first is that things that have been deprecated since 5.0 are finally being pulled. I understand the critics who think that removing . from @INC is insignificant, but I am still for it. I also like that Perl recognizes unhandled merges and dies now. 

Tomorrow, I will be learning about Dancer, Test2 and more with @INC, and visiting with family after

2017/05/17

Contact Me! (If you REALLY need to)

A recent comment from Schlomi Fish said:
Hi! I cannot seem to find any contact information on this page. How should I contact you?
And then linked to a FAQ entry explaining his position on the state of email, comparing the futility of hiding addresses and the benefits of being open.

I have to say, I hadn't thought about this in ... years? In general, I'm active on Twitter (@jacobydave), which is good if you are, but not helpful if you aren't. I try to keep track of the comments, but that doesn't fit every message a person would want to send me.

So, a friendly "Hey, you should put your email on your blog" comment makes sense to me.

But, adding more traffic to the mailbox that friends and relatives have access to doesn't. I'm happy to put up an email address, but I'm less than happy to make it my main email address. It goes to context; my coworkers generally don't get that one either.

I had a long, barely touched by me but used enough by others project that was R syntax highlighting in Komodo Edit, which is dead because it's now native, but the ActiveState packaging used an email address to set the id, so, I created rlangsyntax@gmail.com.

So, to the right, in a section called "More Of Me", there is a requested mailto: pointing to rlangsyntax@gmail.com. I will check it. Use it in good health.

2016/11/19

Graphs are not that Scary!



As with most things I blog about, this starts with Twitter. I follow a lot of people on Twitter, and I use Lists. I want to be able to group people more-or-less on community, because there's the community where they talk about programming, for example, and the community where they talk about music, or the town I live in.

I can begin to break things up myself, but curation is a hard thing, so I wanted to do it automatically. And I spent a long time not knowing what to do. I imagined myself traversing trees in what looks like linked lists reimagined by Cthulhu, and that doesn't sound like much fun at all.

Eventually, I decided to search on "graphs and Perl". Of course, I probably should've done it earlier, but oh well. I found Graph. I had used GD::Graph before, which is a plotting library. (There has to be some index of how overloaded words are.) And once I installed it, I figured it out: As a programmer, all you're dealing with are arrays and hashes. Nothing scary.

Word Ladder


We'll take a problem invented by Lewis Carroll called a "word ladder", where you find your way from one word (for example, "cold") to another ("warm") by changing one letter at a time:

    cold
    coRd
    cArd
    Ward
    warM

Clearly, this can and is often done by hand, but if you're looking to automate it, there are three basic problems: what are the available words, how do you determine when words are one change away, and how do you do this to get the provable shortest path?

First, I went to CERIAS years ago and downloaded word lists. Computer security researchers use them because real words are bad passwords, so, lists of real words can be used to create rainbow tables and the like. My lists are years old, so there may be new words I don't account for, but unlike Lewis Carroll, I can get from APE to MAN in five words, not six.

    ape
    apS
    aAs
    Mas
    maN

Not sure that Lewis Carroll would've accepted AAS, but there you go

There is a term for the number of changes it takes to go from one word to another, and it's called the Levenshtein Distance. I first learned about this from perlbrew, which is how, if you type "perlbrew isntall", it guesses that you meant to type "perlbrew install". It's hardcoded there because perlbrew can't assume you have anything but perl and core modules. I use the function from perlbrew instead of Text::Levenshtein but it is a module worth looking into.

And the final answer is "Put it into a graph and use Dijkstra's Algorithm!"

Perhaps not with the exclamation point.

Showing Code


Here's making a graph of it:

#!/usr/bin/env perl

use feature qw{say} ;
use strict ;
use warnings ;

use Data::Dumper ;
use Graph ;
use List::Util qw{min} ;
use Storable ;

for my $l ( 3 .. 16 ) {
    create_word_graph($l) ;
    }
exit ;

# -------------------------------------------------------------------
# we're creating a word graph of all words that are of length $length
# where the nodes are all words and the edges are unweighted, because
# they're all weighted 1. No connection between "foo" and "bar" because 
# the distance is "3".

sub create_word_graph {
    my $length = shift ;
    my %dict = get_words($length) ;
    my @dict = sort keys %dict ; # sorting probably is unnecessary
    my $g    = Graph->new() ;

    # compare each word to each word. If the distance is 1, put it
    # into the graph. This implementation is O(N**2) but probably
    # could be redone as O(NlogN), but I didn't care to.

    for my $i ( @dict ) {
        for my $j ( @dict ) {
            my $dist = editdist( $i, $j ) ;
            if ( $dist == 1 ) {
                $g->add_edge( $i, $j ) ;
                }
            }
        }

    # Because I'm using Storable to store the Graph object for use
    # later, I only use this once. But, I found there's an endian
    # issue if you try to open Linux-generated Storable files in
    # Strawberry Perl.

    store $g , "/home/jacoby/.word_$length.store" ;
    }

# -------------------------------------------------------------------
# this is where we get the words and only get words of the correct
# length. I have a number of dictionary files, and I put them in
# a hash to de-duplicate them.

sub get_words {
    my $length = shift ;
    my %output ;
    for my $d ( glob( '/home/jacoby/bin/Toys/Dict/*' ) ) {
        if ( open my $fh, '<', $d ) {
            for my $l ( <$fh> ) {
                chomp $l ;
                $l =~ s/\s//g ;
                next if length $l != $length ;
                next if $l =~ /\W/ ;
                next if $l =~ /\d/ ;
                $output{ uc $l }++ ;
                }
            }
        }
    return %output ;
    }

# -------------------------------------------------------------------
# straight copy of Wikipedia's "Levenshtein Distance", straight taken
# from perlbrew. If I didn't have this, I'd probably use 
# Text::Levenshtein.

sub editdist {
    my ( $f, $g ) = @_ ;
    my @a = split //, $f ;
    my @b = split //, $g ;

    # There is an extra row and column in the matrix. This is the
    # distance from the empty string to a substring of the target.
    my @d ;
    $d[ $_ ][ 0 ] = $_ for ( 0 .. @a ) ;
    $d[ 0 ][ $_ ] = $_ for ( 0 .. @b ) ;

    for my $i ( 1 .. @a ) {
        for my $j ( 1 .. @b ) {
            $d[ $i ][ $j ] = (
                  $a[ $i - 1 ] eq $b[ $j - 1 ]
                ? $d[ $i - 1 ][ $j - 1 ]
                : 1 + min( $d[ $i - 1 ][ $j ], $d[ $i ][ $j - 1 ], $d[ $i - 1 ][ $j - 1 ] )
                ) ;
            }
        }

    return $d[ @a ][ @b ] ;
    }

Following are what my wordlists can do. Something tells me that, when we get to 16-letter words, it's more a bunch of disconnected nodes than a graph.

1718 3-letter words
6404 4-letter words
13409 5-letter words
20490 6-letter words
24483 7-letter words
24295 8-letter words
19594 9-letter words
13781 10-letter words
8792 11-letter words
5622 12-letter words
3349 13-letter words
1851 14-letter words
999 15-letter words
514 16-letter words

My solver isn't perfect, and the first thing I'd want to add is ensuring that both the starting and ending words are actually in the word list. Without that, your code goes on forever.

So, I won't show off the whole program below, but it does use Storable, Graph and feature qw{say}.

dijkstra( $graph , 'foo' , 'bar' ) ;

# -------------------------------------------------------------------
# context-specific perl implementation of Dijkstra's Algorithm for
# shortest-path

sub dijkstra {
    my ( $graph, $source, $target, ) = @_ ;

    # the graph pre-exists and is passed in 
    # $source is 'foo', the word we're starting from
    # $target is 'bar', the word we're trying to get to

    my @q ; # will be the list of all words
    my %dist ; # distance from source. $dist{$source} will be zero 
    my %prev ; # this holds our work being every edge of the tree
               # we're pulling from the graph. 

    # we set the the distance for every node to basically infinite, then 
    # for the starting point to zero

    for my $v ( $graph->unique_vertices ) {
        $dist{$v} = 1_000_000_000 ;    # per Wikipeia, infinity
        push @q, $v ;
        }
    $dist{$source} = 0 ;

LOOP: while (@q) {

        # resort, putting words with short distances first
        # first pass being $source , LONG WAY AWAY

        @q = sort { $dist{$a} <=> $dist{$b} } @q ;
        my $u = shift @q ;

        # say STDERR join "\t", $u, $dist{$u} ;

        # here, we end the first time we see the target.
        # we COULD get a list of every path that's the shortest length,
        # but that's not what we're doing here

        last LOOP if $u eq $target ;

        # this is a complex and unreadable way of ensuring that
        # we're only getting edges that contain $u, which is the 
        # word we're working on right now

        for my $e (
            grep {
                my @a = @$_ ;
                grep {/^${u}$/} @a
            } $graph->unique_edges
            ) {

            # $v is the word on the other end of the edge
            # $w is the distance, which is 1 because of the problem
            # $alt is the new distance between $source and $v, 
            # replacing the absurdly high number set before

            my ($v) = grep { $_ ne $u } @$e ;
            my $w   = 1 ;
            my $alt = $dist{$u} + $w ;
            if ( $alt < $dist{$v} ) {
                $dist{$v} = $alt ;
                $prev{$v} = $u ;
                }
            }
        }

    my @nodes = $graph->unique_vertices ;
    my @edges = $graph->unique_edges ;
    return {
        distances => \%dist,
        previous  => \%prev,
        nodes     => \@nodes,
        edges     => \@edges,
        } ;
    }

I return lots of stuff, but the part really necessary is %prev, because that, $source and $target are everything you need. Assuming we're trying to go from FOR to FAR, a number of words will satisfy $prev{FOR}, but it's the one we're wanting. In the expanded case of FOO to BAR, $prev->{BAR} = 'FAR', $prev->{FAR} is 'FOR', and $prev->{FOR} is 'FOO'.

And nothing in there is complex. It's all really hashes or arrays or values. Nothing a programmer should have any problem with.

CPAN has a number of other modules of use: Graph::Dijkstra has that algorithm already written, and Graph::D3 allows you to create a graph in such a way that you can use it in D3.js. Plus, there are a number of modules in Algorithm::* that do good and useful things. So go in, start playing with it. It's deep, there are weeds, but it isn't scary.

2016/10/30

Net::Twitter Cookbook: Favorites and Followers

Favorites.

Also known as "Likes", they're an indication in Twitter that you approve of a status update. Most of the time, they're paired with retweets as signs by the audience to the author that the post is agreeable. Like digital applause.

This is all well and good, but it could be used for so much more, if you had more access and control over them.

So I did.

The first step is to collect them. There's an API to get them, and collecting them in bulk is easy. A problem is avoiding grabbing the same tweet twice.

# as before, the "boilerplate" can be found elsewhere in my blog.
use IO::Interactive qw{ interactive } ;
my $config ;
$config->{start} = 0 ;
$config->{end}   = 200 ;

for ( my $page = $config->{start}; $page <= $config->{end}; ++$page ) {
        say {interactive} qq{\tPAGE $page} ;
        my $r = $twit->favorites( { 
            page => $page ,
            count => 200 ,
            } ) ;
        last unless @$r ;

        # push @favs , @$r ;
        for my $fav (@$r) {
            if ( $config->{verbose} ) { 
                 say {interactive} handle_date( $fav->{created_at} ) 
                 }
            store_tweet( $config->{user}, $fav ) ;
            }
        sleep 60 * 3 ;    # five minutes
        }


Once I had a list of my tweets, one of the first things I did was use them to do "Follow Friday". If you know who you favorited over the last week, it's an easy thing to get a list of the usernames, count them and add them until you have reached the end of the list or 140 characters.

Then, as I started playing with APIs and wanted to write my own ones, I created an API to find ones containing a substring, like json or pizza or sleep. This way, I could begin to use a "favorite" as a bookmark.

(I won't show demo code, because I'm not happy or proud of the the code, which lives in a trailing-edge environment, and because it's more database-related than Twitter-focused.)

As an aside, I do not follow back. There are people who follow me who I have no interest in reading, and there are people I follow who care nothing about my output. In general, I treat Twitter as something between a large IRC client and an RSS reader, and I never expected nor wanted RSS feeds to track me.

But this can be a thing worth tracking, which you can do, without any storage, with the help of a list. Start with getting a list of those following you, those you follow, and the list of accounts (I almost wrote "people", but that isn't guaranteed) in your follows-me list. If they follow you and aren't in your list, add them. If they're in the list and you have started following them, take them out. If they're on the list and aren't following you, drop them. As long as you're not big-time (Twitter limits lists to 500 accounts), that should be enough to keep a Twitter list of accounts you're not following.

use List::Compare ;

    my $list = 'id num of your Twitter list';

    my $followers = $twit->followers_ids() ;
    my @followers = @{ $followers->{ids} } ;

    my $friends = $twit->friends_ids() ;
    my @friends = @{ $friends->{ids} } ;

    my @list = get_list_members( $twit, $list ) ;
    my %list = map { $_ => 1 } @list ;


    my $lc1 = List::Compare->new( \@friends,   \@followers ) ;
    my $lc2 = List::Compare->new( \@friends,   \@list ) ;
    my $lc3 = List::Compare->new( \@followers, \@list ) ;

    # if follows me and I don't follow, put in the list
    say {interactive} 'FOLLOWING ME' ;
    for my $id ( $lc1->get_complement ) {
        next if $list{$id} ;
        add_to_list( $twit, $list, $id ) ;
        }

    # if I follow, take off the list
    say {interactive} 'I FOLLOW' ;
    for my $id ( $lc2->get_intersection ) {
        drop_from_list( $twit, $list, $id ) ;
        }

    # if no longer following me, take off the list
    say {interactive} 'NOT FOLLOWING' ;
    for my $id ( $lc3->get_complement ) {
        drop_from_list( $twit, $list, $id ) ;
        }

#========= ========= ========= ========= ========= ========= =========
sub add_to_list {
    my ( $twit, $list, $id ) = @_ ;
    say STDERR qq{ADDING $id} ;
    eval { $twit->add_list_member(
            { list_id => $list, user_id => $id, } ) ; } ;
    if ($@) {
        warn $@->error ;
        }
    }

#========= ========= ========= ========= ========= ========= =========
sub drop_from_list {
    my ( $twit, $list, $id ) = @_ ;
    say STDERR qq{REMOVING $id} ;
    eval {
        $twit->delete_list_member( { list_id => $list, user_id => $id, } ) ;
        } ;
    if ($@) {
        warn $@->error ;
        }
    }



But are there any you should follow? Are there any posts in the the feed that you might "like"? What do you "like" anyway?

There's a way for us to get an idea of what you would like, which is your past likes. First, we must get, for comparison, a collection of what your Twitter feed is like normally. (I grab 200 posts an hour and store them. This looks and works exactly like my "grab favorites code", except I don't loop it.

    my $timeline = $twit->home_timeline( { count => 200 } ) ;

    for my $tweet (@$timeline) {
        my $id          = $tweet->{id} ;                          # twitter_id
        my $text        = $tweet->{text} ;                        # text
        my $created     = handle_date( $tweet->{created_at} ) ;   # created
        my $screen_name = $tweet->{user}->{screen_name} ;         # user id
        if ( $config->{verbose} ) {
            say {interactive} handle_date( $tweet->{created_at} );
            say {interactive} $text ;
            say {interactive} $created ;
            say {interactive} $screen_name ;
            say {interactive} '' ;
            }
        store_tweet( $config->{user}, $tweet ) ;
        # exit ;
        }


So, we have a body of tweets that you like, and a body of tweets that are a representative sample of what Twitter looks like to you. On to Algorithm::NaiveBayes!

use Algorithm::NaiveBayes ;
use IO::Interactive qw{ interactive } ;
use String::Tokenizer ;

my $list   = 'ID of your list';
my $nb     = train() ;
my @top    = read_list( $config, $nb , $list ) ;

say join ' ' , (scalar @top ), 'tweets' ;

for my $tweet (
    sort { $a->{analysis}->{fave} <=> $b->{analysis}->{fave} } @top ) {
    my $fav = int $tweet->{analysis}->{fave} * 100 ;
    say $tweet->{text} ;
    say $tweet->{user}->{screen_name} ;
    say $tweet->{gen_url} ;
    say $fav ;
    say '' ;
    }

exit ;

#========= ========= ========= ========= ========= ========= =========
# gets the first page of your Twitter timeline.
#
# avoids checking a tweet if it's 1) from you (you like yourself;
#   we get it) and 2) if it doesn't give enough tokens to make a
#   prediction.
sub read_list {
    my $config = shift ;
    my $nb     = shift ;
    my $list   = shift ;

    ...

    my @favorites ;
    my $timeline =     $twit->list_statuses({list_id => $list});

    for my $tweet (@$timeline) {
        my $id          = $tweet->{id} ;                          # twitter_id
        my $text        = $tweet->{text} ;                        # text
        my $created     = handle_date( $tweet->{created_at} ) ;   # created
        my $screen_name = $tweet->{user}->{screen_name} ;         # user id
        my $check       = toke( lc $text ) ;
        next if lc $screen_name eq lc $config->{user} ;
        next if !scalar keys %{ $check->{attributes} } ;
        my $r = $nb->predict( attributes => $check->{attributes} ) ;
        my $fav = int $r->{fave} * 100 ;
        next if $fav < $config->{limit} ;
        my $url = join '/', 'http:', '', 'twitter.com', $screen_name,
            'status', $id ;
        $tweet->{analysis} = $r ;
        $tweet->{gen_url}  = $url ;
        push @favorites, $tweet ;
        }

    return @favorites ;
    }

#========= ========= ========= ========= ========= ========= =========
sub train {

    my $nb = Algorithm::NaiveBayes->new( purge => 1 ) ;
    my $path = '/home/jacoby/.nb_twitter' ;

    # adapted on suggestion from Ken to

    # gets all tweets in your baseline table
    my $baseline = get_all() ;
    for my $entry (@$baseline) {
        my ( $tweet, $month, $year ) = (@$entry) ;
        my $label = join '', $year, ( sprintf '%02d', $month ) ;
        my $ham = toke(lc $tweet) ;
        next unless scalar keys %$ham ;
        $nb->add_instance(
            attributes => $ham->{attributes},
            label      => ['base'],
            ) ;
        }

    gets all tweets in your favorites table
    my $favorites = get_favorites() ;
    for my $entry (@$favorites) {
        my ( $tweet, $month, $year ) = (@$entry) ;
        my $label = join '', $year, ( sprintf '%02d', $month ) ;
        my $ham = toke(lc $tweet) ;
        next unless scalar keys %$ham ;
        $nb->add_instance(
            attributes => $ham->{attributes},
            label      => ['fave'],
            ) ;
        }

    $nb->train() ;
    return $nb ;
    }

#========= ========= ========= ========= ========= ========= =========
# tokenizes a tweet by breaking it into characters, removing URLs
# and short words
sub toke {
    my $tweet = shift ;
    my $ham ;
    my $tokenizer = String::Tokenizer->new() ;
    $tweet =~ s{https?://\S+}{}g ;
    $tokenizer->tokenize($tweet) ;

    for my $t ( $tokenizer->getTokens() ) {
        $t =~ s{\W}{}g ;
        next if length $t < 4 ;
        next if $t !~ /\D/ ;
        my @x = $tweet =~ m{($t)}gmix ;
        $ham->{attributes}{$t} = scalar @x ;
        }
    return $ham ;
    }


Honestly, String::Tokenizer is probably a bit too overkill for this, but I'll go with it for now. It might be better to get a list of the 100 or 500 most common words and exclude them from the tweets, instead of limiting by size. As is, strings like ada and sql would be excluded. But it's good for now.

We get a list of tweets including a number between 0 and 1, representing the likelihood, by Bayes, that I would like the tweet. In the end, it's turned into an integer between 0 and 100. You can also run this against your normal timeline to pull out tweets you would've liked but missed. I often do this

I run the follows_me version on occasion. So far, it is clear to me that the people I don't follow, I don't follow for a reason, and that remains valid.

If you use this and find value in it, please tell me below. Thanks and good coding.

2016/10/28

Using the Symbol Table: "Help"?

I've been looking at command-line code for both fun and work. I know I can have one module handle just the interface, and have the module where the functionality happens pass the functionality along.

#!/usr/bin/env perl

use feature qw'say state' ;
use strict ;
use warnings ;
use utf8 ;

my $w = Wit->new( @ARGV ) ;
$w->run() ;

package Wit ;
use lib '/home/jacoby/lib' ;
use base 'Witter' ;
use Witter::Twitter ;

1;

package Witter ;

# highly adapted from perlbrew.

use feature qw{ say } ;
use strict ;
use warnings ;

sub new {
    my ( $class, @argv ) = @_ ;
    my $self ;
    $self->{foo}  = 'bar' ;
    $self->{args} = [] ;
    if (@argv) {
        $self->{args} = \@argv ;
        }
    return bless $self, $class ;
    }

sub run {
    my ($self) = @_ ;
    $self->run_command( $self->{args} ) ;
    }

sub run_command {
    my ( $self, $args ) = @_ ;

    if (   scalar @$args == 0
        || lc $args->[0] eq 'help'
        || $self->{help} ) {
        $self->help(@$args) ;
        exit ;
        }

    if ( lc $args->[0] eq 'commands' ) {
        say join "\n\t", '', $self->commands() ;
        exit ;
        }

    my $command = $args->[0] ;

    my $s = $self->can("twitter_$command") ;
    unless ($s) {
        $command =~ y/-/_/ ;
        $s = $self->can("twitter_$command") ;
        }

    unless ($s) {

        my @commands = $self->find_similar_commands($command) ;
        if ( @commands > 1 ) {
            @commands = map { '    ' . $_ } @commands ;
            die
                "Unknown command: `$command`. Did you mean one of the following?\n"
                . join( "\n", @commands )
                . "\n" ;
            }
        elsif ( @commands == 1 ) {
            die "Unknown command: `$command`. Did you mean `$commands[0]`?\n"
                ;
            }
        else {
            die "Unknown command: `$command`. Typo?\n" ;
            }
        }

    unless ( 'CODE' eq ref $s ) { say 'Not a valid command' ; exit ; }

    $self->$s(@$args) ;
    }

sub help {
    my ($self,$me,@args) = @_ ;
    say 'HELP!' ;
    say join "\t", @args;
    }

sub commands {
    my ($self) = @_ ;
    my @commands ;

    my $package = ref $self ? ref $self : $self ;
    my $symtable = do {
        no strict 'refs' ;
        \%{ $package . '::' } ;
        } ;

    foreach my $sym ( sort keys %$symtable ) {
        if ( $sym =~ /^twitter_/ ) {
            my $glob = $symtable->{$sym} ;
            if ( defined *$glob{CODE} ) {
                $sym =~ s/^twitter_// ;
                $sym =~ s/_/-/g ;
                push @commands, $sym ;
                }
            }
        }

    return @commands ;
    }

# Some functions removed for sake of brevity


package Witter::Twitter ;

use strict ;
use feature qw{ say state } ;
use warnings FATAL => 'all' ;

use Exporter qw{import} ;
use Net::Twitter ;
use JSON::XS ;


our $VERSION = 0.1 ;

our @EXPORT ;
for my $entry ( keys %Witter::Twitter:: ) {
    next if $entry !~ /^twitter_/mxs ;
    push @EXPORT, $entry ;
    }

sub twitter_foo {
    my ( $self, @args ) = @_ ;
    say "foo" ;
    say join '|', @args ;
    }
1 ;


And the above works when called as below.
jacoby@oz 13:49 60°F 51.24,-112.49 ~ 
$ ./witter 
HELP!


jacoby@oz 13:52 60°F 51.25,-94.51 ~ 
$ ./witter help 
HELP!


jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter commands

 foo

jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter help foo
HELP!
foo

jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter foo
foo
foo

jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter moo
Unknown command: `moo`. Did you mean `foo`?

In the above example, I'm just doing the one add-on module, Witter::Twitter and one function, Witter::Twitter::foo, but clearly, I would want it open-ended, so that if someone wanted to add Witter::Facebook, all the information about the Facebook functions would be in that module.

Then, of course, I would have to use another prefix than twitter_, but we'll leave that, and ensuring that modules don't step on each others' names, to another day.

The part that concerns me is help. Especially help foo. It should be part of the the module it's in; If Witter::Twitter is the module with foo(), only it should be expected to know about foo().

But how to communicate it? I'm flashing on our %docs and $docs{foo}= 'This is foo, silly' but the point of the whole thing is to allow the addition of modules that the initial module doesn't know about, and it would require knowing to look for %Witter::Twitter::docs.

I suppose adding a docs_ function that looks like this.
sub docs_foo {
    return q{
    This explains the use of the 'foo' command 

...
};
END
}


I'm diving into this in part because I have code that uses basically this code, and I need to add functionality to it, and while I'm in there, I might as well make user documentation better. Or even possible.

I'm also parallel-inspired by looking at a Perl project built on and using old Perl ("require 5.005") and recent blog posts about Linus Torvalds and "Good Taste". There's something tasteful about being able to add use Your::Module and nothing else to code, but if the best I can do is knowledge that there's a foo command, with no sense of what it does, that seems like the kind of thing that Linus would rightfully curse me for.

Is there a better way of doing things like this? I recall there being interesting things in Net::Tumblr that would require me to step up and learn Moose or the like. This is yet another important step toward me becoming a better and more tasteful programmer, but not germane to today's ranting.

2016/09/09

Net::Twitter Cookbook: Tweetstorm!

I had planned to get into following, lists and blocking, but a wild hair came over me and I decided I wanted to make a tweetstorm app.

What's a tweetstorm?


And this is the inner loop of it.
    my @tweetstorm_text ; # previously generated. URL shortening and numbering are assumed
    my $screen_name ; #your screen name as text
    my $status_id ; #created but not set, so first tweet won't be a reply-to

    for my $status ( @tweetstorm_text } ) {

        my $tweet ; # the object containing the twee
        $tweet->{status} = $status ;
        $tweet->{in_reply_to_status_id} = $status_id if $status_id ;

        if ( $twit->update($tweet) ) { #if Tweet is successful

            # we get your profile, which contains your most recent tweet
            # $status_id gets set to the ID of that tweet, which then
            # gets used for the next tweet.

            my $profiles
                = $twit->lookup_users( { screen_name => $config->{user} } ) ;
            my $profile = shift @$profiles ;
            my $prev    = $profile->{status} ;
            $status_id = $profile->{status}->{id} ;
            }
        }

So, that's it. I've thrown the whole thing on GitHub if you want to play with it yourself.

2016/09/04

Thoughts on ML Techniques to better handle Twitter

Thinking things through afk and thus not as polished as my normal posts.

Been thinking about grouping my friends (those I follow) strictly by relationship mapping. In part, I haven't done this because I can't read the math in the paper describing it, and in part because there are points where I serve as connecting node between two clusters and they have started interacting independently. I know a joy of Twitter is that it allows people to connect by interest and personality, not geography, but when a programmer in Washington and an activist in Indiana talk food and cats with each other, it makes my "programmer" cluster and my "Indiana" cluster less distinct.

So, what to do?

Topic Modelling.

I know about this via the Talking Machines podcast, and, without mathematic notation, if you take a body of text as a collection of words, the words it contains will vary by subject. If the topic is "politics", the text might contain "vote" and "veto" and "election" and "impeach". If the topic is "football", we'd see "lateral", "quarterback", "tackle" and "touchdown".

Rather than separating Twitter followers into groups simply by interactions, I could start with both certain lists I have curated (and yes, there are both "local tweeters" and "programmer" lists) and hashtags (because if you hashtag your tweet #perl, you likely are talking about Perl) to start to identify what words are more likely to come up when discussing certain subjects, then start adding then to those lists automatically.

If I can work this out 140 characters at a time.

2016/08/19

On Not YET Contributing to an Open Source Project

Not the module in question. 

I had an idea, and the shortest path between here and a working implementation is through CPAN, so I found a module and tried to install it.

No go. Failed tests.

So I find the GitHub repo and make an issue.

That doesn't get me any closer to a working implementation of my idea, nor does it involve me coding or running code. We can't have that, so I took the next step, which was forking, cloning and branching the code, then playing with it to find out what's going on and why.

I won't tell you which repo; that's really besides the point. I've dealt with Perl enough to know that, when the module was last updated, everything was tested and everything worked. There is therefore no condemnation for those who are in CPAN.

Simply speaking, at some point, the API the module is interacting with changed how it works, returning an image instead of JSON that would contain the URL of said image. Not an unreasonable thing to do, I think. That saves you a step, and for my purposes, I'd never have need to get either the image or the location of the image. I feel free to believe that force installing the module would get me
an acceptable outcome.

But if I know a thing is broken and I know how to fix it, and I don't fix it, I'm simply leaving the world in a broken state. That's hardly a responsible response.

The question becomes "Just how do I fix this?", and I see three choices:
  • FOLLOW THE TEST! The test wants an object which is converted from the JSON the API returns. I could easily skip the API call and just return http://example.com/{whatever}/image/. This will even be a valid URL, but when the API changes again, which it will, the URL will be inaccurate. This seems brittle to me.
  • CHANGE THE TEST! The API wants to pass back an image. I can pass that on through and rewrite the test. But the module isn't brand new, which means that someone out there is using the failing function to get the URL of the image and this change will brake that existing code. It's probably already broken; this module won't install, as established, but this is a significant change in the API, which shouldn't be made without consideration. 
  • DROP IT ALL! Remove the function that causes the problem. Remove the tests. Leave only a stub saying "This functionality has been removed". In some ways, this is the coward's way out, but it would be the simplest thing. Easier to remove functionality than to add it.
After I thought about it, I was leaning toward an All-Three approach: Making three branches, implementing each idea, then submitting three parallel pull requests, leaving it to the maintainer to decide the proper choice.

Yeah, that's not a smart plan. 2/3 of that work is going to be not used, by definition. After consultation with my advisors (read, anyone who would listen on Twitter, IRC or Hangouts), I decided that asking before coding anything, so I added details to my (admittedly short and incomplete, submitted before understanding) issue to say "I see three alternatives; what do you want me to do?"

But that is the proper way to do it. It isn't like I went "I want a feature, and here it is." I just want a working module to do the thing I'm thinking of (which might not be as cool as I thought it was), and the things I could do to make it work again could bug the people actually using it. The community of users.

every change breaks someone's workflow
As always, this is a point where xkcd understands and explains all.
So, I have been thinking about this, asking about this and writing about this, rather than implementing the initial idea, or even fixing the problem.

Which frustrates me to no end.

2016/08/12

A Little Fun with OpenCV


I have a webcam that I have set to take pictures at work. Is this a long-term art project? Is it a classic Twitter over-share? An example of Quantified Self gone horribly, horribly wrong? I honestly don't know.

I do know that it's brought me into interesting places before. Getting my camera to work with V4L. Tweeting images. Determining if the screen is locked, both on Unity and Gnome. I've come to the point where I can set @hourly ~/bin/tweet_a_picture.pl. I don't, rather 0 9-18 * * 1-5 ~/bin/tweet_a_picture.pl, because if it's before 9am or on a weekend, I won't be here, so don't even try.

But if I'm pulled away, there are photos like the above. Who needs to see my space without me?

Yeah, who needs to see my space with me? We'll pass on that.

One of the members of Purdue Perl Mongers formerly worked with OpenCV, and I asked him to present on that topic this month. He even created a GitHub repo with his example code. It's Python, not Perl, which will be a digression for this blog, but not an unprecedented one. This gave me enough confidence in OpenCV and how it worked to create a program that uses it to tell if anyone's in the frame.

(The demo code suggested installing python-opencv for Ubuntu, to which I must add that the crucial opencv-data does not come with it. Just a warning.)

# webcam_here.py

# usage: webcam_here.py
# 1

# when runs, returns the number of upper bodies, and thus people, it currently sees

import cv2
# pass either device id (0 usually for webcam) or path to a video file
cam = cv2.VideoCapture(0)

# face cascade bundled with OpenCV
# classifier = cv2.CascadeClassifier('/usr/share/opencv/haarcascades/haarcascade_frontalface_default.xml')
classifier = cv2.CascadeClassifier('/usr/share/opencv/haarcascades/haarcascade_upperbody.xml')
# I found that haarcascade_upperbody did the best for identifying me.

# read in frame
retval, frame = cam.read()

# convert to grayscale
frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

# run classifier
results = classifier.detectMultiScale(
 frame_gray,
 # let's assume face will be close/big if in front of webcam
 minSize=(300,300)
)

# number of found upper bodies in webcam
print len(results)


And here it is in context.
# tweet_a_picture.pl
use 5.010 ;
use strict ;
use warnings ;
use IO::Interactive qw{ interactive } ;
use Net::Twitter ;
use YAML::XS qw{ DumpFile LoadFile } ;
use Getopt::Long ;

use lib '/home/jacoby/lib' ;
use Locked qw{ is_locked there };

my $override = 0 ;

GetOptions(
    'override' => \$override ,
    ) ;

my $config_file = $ENV{ HOME } . '/.twitter.cnf' ;
my $config      = LoadFile( $config_file ) ;

my $user = 'screen_name' ;
my $status = 'Just took a picture' ;

if ( is_locked() == 0 && there() == 1 || $override ) {
    say { interactive } 'unlocked' ;
    # highest supported resolution of my old-school webcam 
    # Twitter DOES have a 5MB limit on file size
    # https://dev.twitter.com/rest/media/uploading-media#imagerecs
    my $cam = q{/usr/bin/fswebcam -q -r 640x480 --jpeg 85 -D 1 --no-banner} ;
    my $time = time ;
    my $image = "/home/jacoby/Pictures/Self/shot-${time}.jpeg" ;
    qx{$cam $image} ;
    tweet_pic( $status , $image ) ;
    say { interactive } $time ;
    }

sub tweet_pic {
    my ( $status , $pic ) = @_ ;
    my ( $access_token, $access_token_secret ) ;
    ( $access_token, $access_token_secret ) = restore_tokens( $user ) ;
    my $twit = Net::Twitter->new(
        traits              => [qw/API::RESTv1_1/],
        consumer_key    => $config->{ consumer_key },
        consumer_secret => $config->{ consumer_secret },
        ssl => 1 ,
        ) ;

    if ( $access_token && $access_token_secret ) {
        $twit->access_token( $access_token ) ;
        $twit->access_token_secret( $access_token_secret ) ;
        }
    unless ( $twit->authorized ) {
        say qq{UNAUTHTORIZED} ; exit ;
        # I really handle this better
        # http://varlogrant.blogspot.com/2016/07/nettwitter-cookbook-how-to-tweet.html
        }

    my $img_ref ;
    my $file = $pic ;
    my ( $filename ) = reverse split m{/} , $pic ;
    push @$img_ref , $file ;
    push @$img_ref , $filename ;

    if ( $twit->update_with_media( $status , $img_ref  ) ) {
        no warnings ;
        say { interactive } $status ;
        }
    else {
        say { interactive } 'FAIL' ;
        }
    }

# is in Locked.pm, but placed here for simplicity
# there are issues with recognizing my webcam, which 
# we quash by sending STDERR to /dev/null
sub there {
    my $checker = '/home/jacoby/bin/webcam_here.py' ;
    return -1 if ! -f $checker ;
    my $o = qx{$checker 2> /dev/null} ;
    chomp $o ;
    return $o ;
    }

sub restore_tokens {
    my ( $user ) = @_ ;
    my ( $access_token, $access_token_secret ) ;
    if ( $config->{ tokens }{ $user } ) {
        $access_token = $config->{ tokens }{ $user }{ access_token } ;
        $access_token_secret =
            $config->{ tokens }{ $user }{ access_token_secret } ;
        }
    return $access_token, $access_token_secret ;
    }

sub save_tokens {
    my ( $user, $access_token, $access_token_secret ) = @_ ;
    $config->{ tokens }{ $user }{ access_token }        = $access_token ;
    $config->{ tokens }{ $user }{ access_token_secret } = $access_token_secret ;
    DumpFile( $config_file, $config ) ;
    return 1 ;
    }


Clearly, this is the start, it's currently very cargo-cult code, and it has several uses. A coming use is to go through my archive of pictures and identifying the ones where I'm gone. That'll either require me getting up to speed with Image::ObjectDetect or Python, but it's a thing I'm willing to do.

An interesting side-issue: I've found that the existing Haar cascades (the XML files that define a thing OpenCV can identify) do not like my glasses, and thus cannot identify my eyes or face with them on, thus, me using the upperbody cascade. I think I should train my own Haar classifier; I know I have enough pics of me for it.

2016/08/06

Net::Twitter Cookbook: Images and Profiles

I've covered how you handle keys and authenticating with Twitter previously, so look into those for more information as we go ahead with sending tweets!

There was a time when Twitter was just text, so you could just send a link to an image. There were sites like TwitPix that hosted images for you, but eventually Twitter developed the ability to host media.

    my $media ;
    push @$media, $config->{ file } ;
    push @$media, 'icon.' . $suffix ;
    if ( $twit->update_with_media( $status , $media ) ) {
        say 'OK' ;
        }

There are four ways you could define the media:

    $media = [ 'path to media' ] ;
    $media = [ 'path to media', 'replacment filename' ] ;
    $media = [ 'path to media', 'replacment filename', 
            Content-Type => mime/type' ] ;
    $media = [  undef , 'replacment filename', 
            Content-Type => mime/type', 'raw media data'] ;


I use the second in order to specify the file name that isn't whatever long and random filename it has. The module has ways to guess the correct mime type, but you can specify to avoid that. The last one, starting with undef, allows you to create or modify an image with Image::Magick or whatever you choose and tweet it without having to involve the filesystem at all.

A few words about what we mean by media. We mean images or video; no audio file option. By images, we mean PNG, JPG, GIF or WEBP. No SVG, and less that 5MB as of this writing. (Check https://dev.twitter.com/ for more info later.)

Video has other constraints, but I don't tweet just video often, if ever. I generally put it to YouTube and post the link.

There's a couple ways you can have more fun with images on Twitter. Those would be with your profile image, the square image that sits next to your tweet, and the profile banner, the landscape image that shows up at the top of your page.

They work like the above, handling media the same way.

    my $image ;
    push @$image, $config->{ file } ;
    push @$image, 'icon.' . $suffix ;
    if ( $twit->update_profile_image( $image ) ) {
        say 'OK' ;
        }

    my $banner ;
    push @$banner, $config->{ file } ;
    push @$banner, 'icon.' . $suffix ;
    if ( $twit->update_profile_banner( $banner ) ) {
        say 'OK' ;
        }


Profile images are square, and are converted to square if your initial image is not square. For both, JPG, GIF or PNG are allowed and are converted upon upload. If you try to upload an animated GIF, they will use the first frame. This can make your profile image less fun, but if you have a feed full of pictures that throb and spin, that can make you feel seasick.

And, since we're hitting profile-related issues, perhaps I should handle your profiles and we can get this behind us.

Changing your username, going from @jacobydave to something else, is doable but not via the API. It isn't a thing you can or should be doing with an API. You can change other details, however. You can change your name ("Dave Jacoby"), description ("Guy blogging about Perl's Net::Twitter library. Thought Leader. Foodie."), website ("http://varlogrant.blogspot.com/") and location ("Dark Side of the Moon").

    use Getopt::Long ;

    my $config ;
    GetOptions(
        'description=s' => \$config->{ description },
        'location=s'    => \$config->{ location },
        'name=s'        => \$config->{ name },
        'web=s'         => \$config->{ url },
        ) ;

    my $params ; for my $p ( qw{ name url location description } ) {
    $params->{ $p } = $config->{ $p } if $config->{ $p } ; }
    $params->{ include_entities } = 0 ;  $params->{ skip_status } = 1
    ;

    if ( $twit->update_profile( $params ) ) {
        say 'OK' ;
        }
    }


I *DO* have fun fun with location, setting it with a number of strings; "My Happy Place" on Saturday, "Never got the hang..." for Thursdays. You can set this to any string, and Twitter does nearly nothing with it. I'd suggest you ignore it or set it to something general or clever, and forget it.

Now that we've covered how you tweet and handle your identity a little, next time we'll start to get into relationships, which are the things that make a social network social.

2016/08/02

Net::Twitter Cookbook: How I tweet, plus

Previously, I wrote a post showing the basics on how to send a tweet using Perl and Net::Twitter. I showed the easiest you can do it.

Below is the code that I use most often. It is a command-line tool, where it's used something along the lines of named_twitter.pl jacobydave text of my tweet. Except, that's not how I type it, thanks to my .alias file. I have twitter aliased to '~/bin/named_twitter jacobydave ' and normally tweet like twitter this is how I tweet.

This isn't to say I never automate tweets; I certainly do. But it is a rare part of what I do with the Twitter API and Net::Twitter. I will dive deeper into issues with tweeting and direct messages, both in a technical and social matter, in a later post.

But I said that you have the consumer key and secret, which identify you as a service, and the access token and secret, which identify you as a user of the service. In the above code sample, I use YAML to store this data, but you could use JSON, Sqlite or anything else to store it. Among other benefits, you can put your code into GitHub, as I did above, without exposing anything.

As I proceed, I will assume that tokens are handled somehow and proceed directly to the cool part.

Again, you get the consumer key by going to apps.twitter.com and creating a new app.


You log into apps.twitter.com with your Twitter account, click "Create New App" and fill in the details. When I created my three, you had to specifically choose "can send DMs" specifically, so if you're creating apps to follow along, I do suggest you allow yourself that option.

2016/07/27

Net::Twitter Cookbook: How to Tweet



The first line between Twitter Use and Twitter Obsession is TweetDeck. That's when the update-on-demand single-thread of the web page gives way to multiple constantly-updated streams of the stream-of-consciousness ramblings of the Internet.

That's the first line.

The second line between Twitter use and Twitter obsession is when you want to automate the work. If you're an R person, that's twitteR. If you work in Python, that's tweepy.

And, if you're like me, and you normally use Perl, we're talking Net::Twitter.

What follows is the simplest possible Net::Twitter program.

#!/usr/bin/env perl
use feature qw{ say } ;
use strict ;
use warnings ;
use Net::Twitter ;

# the consumer key and secret identify you as a service. 
# you register your service at https://apps.twitter.com/
# and receive the key and secret

# you really don't want to have these written into your script

my $consumer_key    = 'ckckckckckckckckckckck' ;
my $consumer_secret = 'cscscscscscscscscscscscscscscscscscscscs' ;

my $twit = Net::Twitter->new(
    traits          => [qw/API::RESTv1_1/],
    consumer_key    => $consumer_key,
    consumer_secret => $consumer_secret,
    ssl             => 1,
    ) ;

# the access token and secret identify you as a user.
# the registration process takes place below.
# the first time you run this program, you will not be authorized,
# and the program will give you a URL to open in a browser where
# you are already logged into twitter.

# you really don't want to have these written into your script

my $access_token = '1111111111-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' ;
my $access_token_secret = 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz' ;

$twit->access_token($access_token) ;
$twit->access_token_secret($access_token_secret) ;

# everything interesting will occur inside this if statement
if ( $twit->authorized ) {
    if ( $twit->update('Hello World!') ) {
        say 'It worked!' ;
        }
    else {
        say 'Fail' ;
        }
    }
else {
    # You have no auth token
    # go to the auth website.
    # they'll ask you if you wanna do this, then give you a PIN
    # input it here and it'll register you.
    # then save your token vals.

    say "Authorize this app at ", $twit->get_authorization_url,
        ' and enter the PIN#' ;
    my $pin = <stdin> ;    # wait for input
    chomp $pin ;
    my ( $access_token, $access_token_secret, $user_id, $screen_name ) =
        $twit->request_access_token( verifier => $pin ) ;

    say 'The following lines need to be copied and pasted above' ;
    say $access_token ;
    say $access_token_secret ;
    }

Again, this is as simple as we can reasonably do, without pulling the keys into a separate file, which I, again, strongly recommend you do. (I personally use YAML as the way I store and restore data such as access tokens and consumer keys. I will demonstrate that in a later post.)

2016/06/08

Quantified Self: For What?


This is my daily step count since I first got a FitBit in 2012, in handy heatmap form.

It shows that 2014 was a pretty active year.

It shows that this year, I've really fallen off the game.

It shows that the main purpose of this process for me, of learning how to grab the data and plot it in different and hopefully useful ways, has succeeded.

It shows, really, that I'm much more about collecting the data than using it to change my life.

And I can only see that as a failure.

I've built other things on top of this. My daily steps pop up in my Twitter feed and bash prompt. If I my battery gets low, I get notified on my tablet. If I go several days without a connection (if the battery dies without me noticing, or if I lose it, as I have done recently), I also get notified. I've made it very convenient to me.

But I failed to make greater amounts of movement an important part of my life. I failed to develop an appreciation for running or walking, at least in comparison to everything else I do.

So, I need to start thinking about how I can change my behavior.

And I probably shouldn't get a replacement FitBit until I have a plan for that.

2015/10/17

Long, Boring To-Do Post

I'm considering stepping up my web presence, and as a precursor to that, I've created a twitter account specifically connected to this blog, @varlogrant. So, I need to do things to make it look more like I'm using it and less like it's an egg. (If I made the account picture the cover of that one Wilco album, would people get the joke?)

I certainly can automate a lot of stuff, going through the back catalog and retweeting old posts, but the question is, how much of that is just spammy? And, to what extent should I move my opinionated tech tweeting from @jacobydave to @varlogrant?

Beyond that, it strikes me that blogs where the blogger is more-or-less talking to himself are self-limiting, so I should start blogging more about certain subjects and less about things that are annoying me right this minute. 

That being said:
  • I am involved in a group creating an annual event. Specifically, I'm the web guy. There are some administrivia things going on, creating the pages for the team. This is a small matter of Wordpress time, so not hard. 
  • A harder thing is this: We have photos of previous presenters, which were existing head-shots of them, from before their presentations. We also have a large library of photos from the events. I've decided that the smart move is to use Flickr's API and WebService::Simple to grab all the old photos, use smartcrop.js to crop them to the appropriate size, and either personally chose a good one or make a web tool to crowdsource that amongst the group. This process seems more fun to me than the other thing.
  • I promised a while ago to contribute some R-specific stuff to Bobby Tables, and have done jack since. I made some progress on it recently, but need to install a more modern version of R to do appropriate testing before I make a PR. When I first looked into it, I saw no SQL escaping and no placeholders, but now I'm seeing some progress. Nothing's quite up to snuff, in my opinion, but it's better. 
  • A side project I'm involved in has need of Bluetooth Low Energy support, and I've done the slightest poking with it. I need to do more. It seems that a few necessary tools for debugging are Unix/Linux/Mac only, and my dev laptop is Windows, so I need to either get going with VirtualBox, figure things out in Visual Studio or let it go.
  • There's also need for a smartphone app, and my experiences with Eclipse and Android Studio haven't been pleasant. I know there's Cordova integration with Visual Studio, so that seems to be the quick way in. I don't know if I can do any BLE stuff within a Cordova app, but we'll get there when we get there.
  • There's another side project I'm involved in, called Lafayettech. Specifically, I'm in the proto-makerspace corner, Lafayettech Labs. And it seems like I'm the only one involved in it. So, I am thinking of stopping. Right now, there's a few self-perpetuating scripts in a crontab file that do much of the work. I need to decide something about this.
There's a few more things that should be here, but I don't have together enough to even make a lame bullet point for.

2015/09/14

Thoughts on Machine Learning and Twitter Tools

I have a lot of Twitter data.

I decided a few months ago to get my list of those I follow and those following me, and then find that same information about each of them.

This took quite some time, as Twitter stops you from getting too much at a time.

I found a few interesting things. One was, of all the people I follow and all following me, I had the highest number of bi-directional connections (I follow them, they follow me; let's say "friends") of any, at something like 150.

But I'm thinking about writing the next step, and I gotta say, I don't wanna.

In part, that data's dead. It's Spring me, and my Spring followers. It's Fall now, and I don't know that the data is valid anymore.

So I'm thinking about the next step.

If "Big Data" is more than a buzzword, it is working with data that never ends. I mean, my lab has nodes in a huge research cluster, that contain like a quarter-TB of RAM. That's still a mind-boggling thing when I think about it. And it's not like we let that sit empty; we use it. Well, I don't. My part of the workflow is when the users bring their data and, increasingly, when it's done and being sent on. It clearly is large data sets being handled, but it isn't "Big Data", because we finish it, package it, and give it to our users.

"Big Data", it seems to me, means you find out more now than you did before, that you're always adding to it, because it never ends. "The Torture Never Stops", as Zappa said.

In the case of Twitter, I can get every tweet in my feed and every tweet I favorite. I could write that thing and have it run regularly, starting now. Questions: What do I do? How do I do it?

Here's the Why:

There's tools out there that I don't understand, and they are powerful tools. I want to understand them. I want to write things with these things that interest me, because it's an uncommon day that I'm interested in what I code most of the time.

Then there's Twitter. You can say many many things about Twitter, and it's largely true. But, there are interesting people talking about the interesting things they do, and and other people retweeting it. With luck, I can draw from that, create connections, learn things and do interesting things.

So, while there is the "make a thing because a thing" part, I'm hoping to use this.

So, the What:

A first step is to take my Favorites data and use it as "ham" in a Bayesian way to tell what kind of thing I like. I might need to come up with a mechanism beyond unfollowing to indicate what I don't want to see. Maybe do a search on "GamerGate", "Football" and "Cats"?

Anyway, once that's going, let it loose on my feed and have it bubble up tweets that it says I might favorite, that "fit the profile", so to speak. I'm thinking my next step is installing a Bayesian library like Algorithm::Bayesian or Algorithm::NaiveBayes, running my year+ worth of Twitter favorites as ham, and have it tell me once a day the things I should've favorited but didn't. Once I have that, I'll reorient and go from there.

2015/08/21

Perl Module for interacting with Globus

Ever heard of Globus?

No?

I'm kinda not surprised. It's a bit of an obscure thing.

Imagine you have research data. A collaborator wants to analyse it. Thing is, you're where you are and the collaborator is on the other side of the continent. There are many ways to move things, but many of them involve you having to be fairly technically sophisticated in order to make them work. Or your collaborator, who is an expert in your domain but not necessarily in file transfer. Plus, there's permissions and security and all that.

Or, you can use Globus.

We're changing how we handle data sharing here in the lab. We just moved file systems, moving to one that supports access control lists (ACLs), which means that, instead of playing games with symbolic links in order to share a researcher's work, we could simply setfacl -R -m 'u:user:r-x' and go from there.

That was fun, but because an ACL has to be set for every file, changes take a long time, so testing takes a long time, and updates take a long time. But it means that local researchers have access on the Big Iron.

But not everyone we want to share with is local. Some are on the other side of the continent. Thus Globus.

And, soon, Net::Globus.

It uses their first API, which is exposed via SSH. This is not my preferred interface, but I was able to get it to do the subset I needed fast enough, and I'll go back and make a thing work with their REST API in my copious free time. I mean, I could just use the Python API, right?

If you were to take my Globus.pm and copy it into your ~/lib directory, and you had the right stuff set up with SSH keys, it'd work. For the subset that isn't stubtastic, it's there.

I started of this project with dzil new, which is as much of Dist::Zilla as I know how to use. (I didn't start this paragraph intending an Elvis Costello reference, but there you go.) I know how to use git, how to make pull requests and merge them. My contribution graph is more white than green, but GitHub is not integral to my workflow, containing more my toys than my work. So, I'm pretty close to OK for the managing of a repo, but the steps toward getting it onto CPAN will involve me reading and asking a lot of stupid questions irc.perl.org.

Plus, the tests. Oh, man, the tests....

Perl is good because Perl is tested, and the tests I know how to write are tests I consider stupid. So, I expect to spend some time in chromatic's book of testing and in the tests for Net::Twitter and the like, trying to figure out what needs to be tested and how to write them.

So, I'm not ready to put it up to CPAN yet, but I see the path, because I finally have a module I could imagine someone else needing.

So, if this is a thing you think you have a use for, try it. If you think I did something stupid, I'm sure I did. Suggestions are great; pull requests are better. And thanks.

2015/05/18

Some Wisdom From Twitter

2015/04/01

Thinking through Git in my work

This is mostly me thinking through some issues.

I use git and GitHub a fair amount. Not enough, not deeply enough, but a fair amount.

I understand Use Case #1, Compiled App, like RLangSyntax:

  • code in ~/dev/Project
  • subdirs (test/, lib/, examples/, docs/, etc) and documentation, license and build scripts in to git
  • git push origin master
  • compile code to binary, deploy the binary elsewhere (like GitHub's releases tab)
When someone wants to use this code, git clone Project gets you all you need to build it, except for compilers and associated libraries, which should be in README.md. (I forgot to put how to build into the RLangSyntax docs, then forgot how to build RLangSyntax. Forgive me, Admin, for I have sinned.)

Let's jump to Use Case #2, Perl Library, like Spark.pm.
  • code in ~/dev/Project
  • no build instructions; it's built
  • tests in ~/dev/Project/t/spark.t and the like (which this doesn't have, to my shame)
  • git push origin master
This is where I get confused. Eventually, this code needs to be in /production/lib, but you don't want to deploy using git pull or git clone, because you don't want /production/lib/Project/. Or, maybe you do and I just don't get it. Still, this is a case that I can do an acceptably wrong thing as required.

Use Case #3 is Perl command-line tools. We'll take on my twitter tool.
  • code in ~/dev/project
  • git push origin master
This begs about the same question as Perl Library. It could work to have ~production/bin/twitter.pl/twitter.pl, but then you have to expand your path to include every little thing. It gets more involved if you have libraries with executables in the repo, or the reverse, but let's get to the real hairball.

Use Case #4, the Hairball, is our web stuff.

  • ~web/ - document root
  • ~web/data - holds data that we want to make web-accessable
  • ~web/images - images for page headers and other universal things
  • ~web/lib - all our JavaScript
  • ~web/css - all our Cascading Style Sheets
  • ~web/cgi-bin - our server-side code
So, we might have the server-side code in ~web/cgi-bin/service/Project/foo.cgi, the client-side code in ~web/lib/Project/foo.js but maybe in ~web/lib/foo.js, the style in ~web/css/project.css and ~/web/css/service.css, and of course the libraries in ~production/lib/.

Maybe the key is to think of the ~web/lib and ~web/css as variations of Use Case #2, but the problem is that a lot of my JS code isn't general like the Perl code. I mean, wherever you want a sparkline, you can use Spark.pm, but the client code for ~/cgi-bin/service/Project/foo.cgi is going to mostly be very specific to foo.cgi except for the jQuery, Mustache and a few other things that are in use across services and projects.

A possible solution, having things be in ~web/service, with the JSON APIs in ~web/cgi-bin/service/apis and the JavaScript in either ~web/service/lib or ~/web/lib depending on how universal it is, but we lose certain abilities. I certainly have written CGI code which mostly puts out as little HTML as it needs to get the JS, which works for the small audience those tools need.

I mostly code foo.js in relation to foo.cgi or foo.html, but making tests and breaking it into pieces may keep me from having KLOC-size libraries I hate to work on in the future.

And here, we have departed from git into project and web architecture, and into best practices. Still, any suggestions would be gladly accepted.