Net::Twitter Cookbook: Favorites and Followers

Favorites.

Also known as "Likes", they're an indication in Twitter that you approve of a status update. Most of the time, they're paired with retweets as signs by the audience to the author that the post is agreeable. Like digital applause.

This is all well and good, but it could be used for so much more, if you had more access and control over them.

So I did.

The first step is to collect them. There's an API to get them, and collecting them in bulk is easy. A problem is avoiding grabbing the same tweet twice.

# as before, the "boilerplate" can be found elsewhere in my blog.
use IO::Interactive qw{ interactive } ;
my $config ;
$config->{start} = 0 ;
$config->{end}   = 200 ;

for ( my $page = $config->{start}; $page <= $config->{end}; ++$page ) {
        say {interactive} qq{\tPAGE $page} ;
        my $r = $twit->favorites( { 
            page => $page ,
            count => 200 ,
            } ) ;
        last unless @$r ;

        # push @favs , @$r ;
        for my $fav (@$r) {
            if ( $config->{verbose} ) { 
                 say {interactive} handle_date( $fav->{created_at} ) 
                 }
            store_tweet( $config->{user}, $fav ) ;
            }
        sleep 60 * 3 ;    # five minutes
        }

Once I had a list of my tweets, one of the first things I did was use them to do "Follow Friday". If you know who you favorited over the last week, it's an easy thing to get a list of the usernames, count them and add them until you have reached the end of the list or 140 characters.

Then, as I started playing with APIs and wanted to write my own ones, I created an API to find ones containing a substring, like json or pizza or sleep. This way, I could begin to use a "favorite" as a bookmark.

(I won't show demo code, because I'm not happy or proud of the the code, which lives in a trailing-edge environment, and because it's more database-related than Twitter-focused.)

As an aside, I do not follow back. There are people who follow me who I have no interest in reading, and there are people I follow who care nothing about my output. In general, I treat Twitter as something between a large IRC client and an RSS reader, and I never expected nor wanted RSS feeds to track me.

But this can be a thing worth tracking, which you can do, without any storage, with the help of a list. Start with getting a list of those following you, those you follow, and the list of accounts (I almost wrote "people", but that isn't guaranteed) in your follows-me list. If they follow you and aren't in your list, add them. If they're in the list and you have started following them, take them out. If they're on the list and aren't following you, drop them. As long as you're not big-time (Twitter limits lists to 500 accounts), that should be enough to keep a Twitter list of accounts you're not following.

use List::Compare ;

    my $list = 'id num of your Twitter list';

    my $followers = $twit->followers_ids() ;
    my @followers = @{ $followers->{ids} } ;

    my $friends = $twit->friends_ids() ;
    my @friends = @{ $friends->{ids} } ;

    my @list = get_list_members( $twit, $list ) ;
    my %list = map { $_ => 1 } @list ;


    my $lc1 = List::Compare->new( \@friends,   \@followers ) ;
    my $lc2 = List::Compare->new( \@friends,   \@list ) ;
    my $lc3 = List::Compare->new( \@followers, \@list ) ;

    # if follows me and I don't follow, put in the list
    say {interactive} 'FOLLOWING ME' ;
    for my $id ( $lc1->get_complement ) {
        next if $list{$id} ;
        add_to_list( $twit, $list, $id ) ;
        }

    # if I follow, take off the list
    say {interactive} 'I FOLLOW' ;
    for my $id ( $lc2->get_intersection ) {
        drop_from_list( $twit, $list, $id ) ;
        }

    # if no longer following me, take off the list
    say {interactive} 'NOT FOLLOWING' ;
    for my $id ( $lc3->get_complement ) {
        drop_from_list( $twit, $list, $id ) ;
        }

#========= ========= ========= ========= ========= ========= =========
sub add_to_list {
    my ( $twit, $list, $id ) = @_ ;
    say STDERR qq{ADDING $id} ;
    eval { $twit->add_list_member(
            { list_id => $list, user_id => $id, } ) ; } ;
    if ($@) {
        warn $@->error ;
        }
    }

#========= ========= ========= ========= ========= ========= =========
sub drop_from_list {
    my ( $twit, $list, $id ) = @_ ;
    say STDERR qq{REMOVING $id} ;
    eval {
        $twit->delete_list_member( { list_id => $list, user_id => $id, } ) ;
        } ;
    if ($@) {
        warn $@->error ;
        }
    }

But are there any you should follow? Are there any posts in the the feed that you might "like"? What do you "like" anyway?

There's a way for us to get an idea of what you would like, which is your past likes. First, we must get, for comparison, a collection of what your Twitter feed is like normally. (I grab 200 posts an hour and store them. This looks and works exactly like my "grab favorites code", except I don't loop it.

    my $timeline = $twit->home_timeline( { count => 200 } ) ;

    for my $tweet (@$timeline) {
        my $id          = $tweet->{id} ;                          # twitter_id
        my $text        = $tweet->{text} ;                        # text
        my $created     = handle_date( $tweet->{created_at} ) ;   # created
        my $screen_name = $tweet->{user}->{screen_name} ;         # user id
        if ( $config->{verbose} ) {
            say {interactive} handle_date( $tweet->{created_at} );
            say {interactive} $text ;
            say {interactive} $created ;
            say {interactive} $screen_name ;
            say {interactive} '' ;
            }
        store_tweet( $config->{user}, $tweet ) ;
        # exit ;
        }

So, we have a body of tweets that you like, and a body of tweets that are a representative sample of what Twitter looks like to you. On to Algorithm::NaiveBayes!

use Algorithm::NaiveBayes ;
use IO::Interactive qw{ interactive } ;
use String::Tokenizer ;

my $list   = 'ID of your list';
my $nb     = train() ;
my @top    = read_list( $config, $nb , $list ) ;

say join ' ' , (scalar @top ), 'tweets' ;

for my $tweet (
    sort { $a->{analysis}->{fave} <=> $b->{analysis}->{fave} } @top ) {
    my $fav = int $tweet->{analysis}->{fave} * 100 ;
    say $tweet->{text} ;
    say $tweet->{user}->{screen_name} ;
    say $tweet->{gen_url} ;
    say $fav ;
    say '' ;
    }

exit ;

#========= ========= ========= ========= ========= ========= =========
# gets the first page of your Twitter timeline.
#
# avoids checking a tweet if it's 1) from you (you like yourself;
#   we get it) and 2) if it doesn't give enough tokens to make a
#   prediction.
sub read_list {
    my $config = shift ;
    my $nb     = shift ;
    my $list   = shift ;

    ...

    my @favorites ;
    my $timeline =     $twit->list_statuses({list_id => $list});

    for my $tweet (@$timeline) {
        my $id          = $tweet->{id} ;                          # twitter_id
        my $text        = $tweet->{text} ;                        # text
        my $created     = handle_date( $tweet->{created_at} ) ;   # created
        my $screen_name = $tweet->{user}->{screen_name} ;         # user id
        my $check       = toke( lc $text ) ;
        next if lc $screen_name eq lc $config->{user} ;
        next if !scalar keys %{ $check->{attributes} } ;
        my $r = $nb->predict( attributes => $check->{attributes} ) ;
        my $fav = int $r->{fave} * 100 ;
        next if $fav < $config->{limit} ;
        my $url = join '/', 'http:', '', 'twitter.com', $screen_name,
            'status', $id ;
        $tweet->{analysis} = $r ;
        $tweet->{gen_url}  = $url ;
        push @favorites, $tweet ;
        }

    return @favorites ;
    }

#========= ========= ========= ========= ========= ========= =========
sub train {

    my $nb = Algorithm::NaiveBayes->new( purge => 1 ) ;
    my $path = '/home/jacoby/.nb_twitter' ;

    # adapted on suggestion from Ken to

    # gets all tweets in your baseline table
    my $baseline = get_all() ;
    for my $entry (@$baseline) {
        my ( $tweet, $month, $year ) = (@$entry) ;
        my $label = join '', $year, ( sprintf '%02d', $month ) ;
        my $ham = toke(lc $tweet) ;
        next unless scalar keys %$ham ;
        $nb->add_instance(
            attributes => $ham->{attributes},
            label      => ['base'],
            ) ;
        }

    gets all tweets in your favorites table
    my $favorites = get_favorites() ;
    for my $entry (@$favorites) {
        my ( $tweet, $month, $year ) = (@$entry) ;
        my $label = join '', $year, ( sprintf '%02d', $month ) ;
        my $ham = toke(lc $tweet) ;
        next unless scalar keys %$ham ;
        $nb->add_instance(
            attributes => $ham->{attributes},
            label      => ['fave'],
            ) ;
        }

    $nb->train() ;
    return $nb ;
    }

#========= ========= ========= ========= ========= ========= =========
# tokenizes a tweet by breaking it into characters, removing URLs
# and short words
sub toke {
    my $tweet = shift ;
    my $ham ;
    my $tokenizer = String::Tokenizer->new() ;
    $tweet =~ s{https?://\S+}{}g ;
    $tokenizer->tokenize($tweet) ;

    for my $t ( $tokenizer->getTokens() ) {
        $t =~ s{\W}{}g ;
        next if length $t < 4 ;
        next if $t !~ /\D/ ;
        my @x = $tweet =~ m{($t)}gmix ;
        $ham->{attributes}{$t} = scalar @x ;
        }
    return $ham ;
    }

Honestly, String::Tokenizer is probably a bit too overkill for this, but I'll go with it for now. It might be better to get a list of the 100 or 500 most common words and exclude them from the tweets, instead of limiting by size. As is, strings like ada and sql would be excluded. But it's good for now.

We get a list of tweets including a number between 0 and 1, representing the likelihood, by Bayes, that I would like the tweet. In the end, it's turned into an integer between 0 and 100. You can also run this against your normal timeline to pull out tweets you would've liked but missed. I often do this

I run the follows_me version on occasion. So far, it is clear to me that the people I don't follow, I don't follow for a reason, and that remains valid.

If you use this and find value in it, please tell me below. Thanks and good coding.

Using the Symbol Table: "Help"?

I've been looking at command-line code for both fun and work. I know I can have one module handle just the interface, and have the module where the functionality happens pass the functionality along.

#!/usr/bin/env perl

use feature qw'say state' ;
use strict ;
use warnings ;
use utf8 ;

my $w = Wit->new( @ARGV ) ;
$w->run() ;

package Wit ;
use lib '/home/jacoby/lib' ;
use base 'Witter' ;
use Witter::Twitter ;

1;

package Witter ;

# highly adapted from perlbrew.

use feature qw{ say } ;
use strict ;
use warnings ;

sub new {
    my ( $class, @argv ) = @_ ;
    my $self ;
    $self->{foo}  = 'bar' ;
    $self->{args} = [] ;
    if (@argv) {
        $self->{args} = \@argv ;
        }
    return bless $self, $class ;
    }

sub run {
    my ($self) = @_ ;
    $self->run_command( $self->{args} ) ;
    }

sub run_command {
    my ( $self, $args ) = @_ ;

    if (   scalar @$args == 0
        || lc $args->[0] eq 'help'
        || $self->{help} ) {
        $self->help(@$args) ;
        exit ;
        }

    if ( lc $args->[0] eq 'commands' ) {
        say join "\n\t", '', $self->commands() ;
        exit ;
        }

    my $command = $args->[0] ;

    my $s = $self->can("twitter_$command") ;
    unless ($s) {
        $command =~ y/-/_/ ;
        $s = $self->can("twitter_$command") ;
        }

    unless ($s) {

        my @commands = $self->find_similar_commands($command) ;
        if ( @commands > 1 ) {
            @commands = map { '    ' . $_ } @commands ;
            die
                "Unknown command: `$command`. Did you mean one of the following?\n"
                . join( "\n", @commands )
                . "\n" ;
            }
        elsif ( @commands == 1 ) {
            die "Unknown command: `$command`. Did you mean `$commands[0]`?\n"
                ;
            }
        else {
            die "Unknown command: `$command`. Typo?\n" ;
            }
        }

    unless ( 'CODE' eq ref $s ) { say 'Not a valid command' ; exit ; }

    $self->$s(@$args) ;
    }

sub help {
    my ($self,$me,@args) = @_ ;
    say 'HELP!' ;
    say join "\t", @args;
    }

sub commands {
    my ($self) = @_ ;
    my @commands ;

    my $package = ref $self ? ref $self : $self ;
    my $symtable = do {
        no strict 'refs' ;
        \%{ $package . '::' } ;
        } ;

    foreach my $sym ( sort keys %$symtable ) {
        if ( $sym =~ /^twitter_/ ) {
            my $glob = $symtable->{$sym} ;
            if ( defined *$glob{CODE} ) {
                $sym =~ s/^twitter_// ;
                $sym =~ s/_/-/g ;
                push @commands, $sym ;
                }
            }
        }

    return @commands ;
    }

# Some functions removed for sake of brevity

package Witter::Twitter ;

use strict ;
use feature qw{ say state } ;
use warnings FATAL => 'all' ;

use Exporter qw{import} ;
use Net::Twitter ;
use JSON::XS ;


our $VERSION = 0.1 ;

our @EXPORT ;
for my $entry ( keys %Witter::Twitter:: ) {
    next if $entry !~ /^twitter_/mxs ;
    push @EXPORT, $entry ;
    }

sub twitter_foo {
    my ( $self, @args ) = @_ ;
    say "foo" ;
    say join '|', @args ;
    }
1 ;

And the above works when called as below.

jacoby@oz 13:49 60°F 51.24,-112.49 ~ 
$ ./witter 
HELP!


jacoby@oz 13:52 60°F 51.25,-94.51 ~ 
$ ./witter help 
HELP!


jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter commands

 foo

jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter help foo
HELP!
foo

jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter foo
foo
foo

jacoby@oz 13:53 60°F 50.59,-88.64 ~ 
$ ./witter moo
Unknown command: `moo`. Did you mean `foo`?

In the above example, I'm just doing the one add-on module, Witter::Twitter and one function, Witter::Twitter::foo, but clearly, I would want it open-ended, so that if someone wanted to add Witter::Facebook, all the information about the Facebook functions would be in that module.

Then, of course, I would have to use another prefix than twitter_, but we'll leave that, and ensuring that modules don't step on each others' names, to another day.

The part that concerns me is help. Especially help foo. It should be part of the the module it's in; If Witter::Twitter is the module with foo(), only it should be expected to know about foo().

But how to communicate it? I'm flashing on our %docs and $docs{foo}= 'This is foo, silly' but the point of the whole thing is to allow the addition of modules that the initial module doesn't know about, and it would require knowing to look for %Witter::Twitter::docs.

I suppose adding a docs_ function that looks like this.

sub docs_foo {
    return q{
    This explains the use of the 'foo' command 

...
};
END
}

I'm diving into this in part because I have code that uses basically this code, and I need to add functionality to it, and while I'm in there, I might as well make user documentation better. Or even possible.

I'm also parallel-inspired by looking at a Perl project built on and using old Perl ("require 5.005") and recent blog posts about Linus Torvalds and "Good Taste". There's something tasteful about being able to add use Your::Module and nothing else to code, but if the best I can do is knowledge that there's a foo command, with no sense of what it does, that seems like the kind of thing that Linus would rightfully curse me for.

Is there a better way of doing things like this? I recall there being interesting things in Net::Tumblr that would require me to step up and learn Moose or the like. This is yet another important step toward me becoming a better and more tasteful programmer, but not germane to today's ranting.

Gender and Wearables?

#wearables etc (arduino, others) is the rare tech where women are present and often set the pace of the tech. hardware more welcoming? ;)

— Su-Shee (@sheeshee) July 7, 2015

First I heard about modern wearables was at Indiana Linuxfest, where the speaker went on about the coming wave of microcontrollers and posited a belt buckle that was aware of when it was pointing toward magnetic north and activate a haptic feedback device, so that, for the wearer, eventually true sense of direction would eventually become another sense.

I'm sure I could find a sensor that could tell me that, that could ship from China and cost me less than a gumball. I'm sure I could easily get a buzzer, that I could control it all with a small Arduino board like a Trinket or Flora or Nano or the like. and Jimmy DiResta has already taught me how to make a belt buckle. And I actually kinda want one. But I haven't made it.

In part it's because my available resources to push toward projects like this are small at the moment. In part, though, it's because, once I put on my watch, my tablet, my glasses and the Leatherman on my belt, I'm accessorized out.

I think most American men are about the same.