Cookie Notice

As far as I know, and as far as I remember, nothing in this page does anything with Cookies.
Showing posts sorted by relevance for query Perl. Sort by date Show all posts
Showing posts sorted by relevance for query Perl. Sort by date Show all posts

2016/11/08

Modern Perl but not Modern::Perl

This started while driving to work. If I get mail from coworkers, I get Pushover notifications, and halfway from home, I got a bunch of notifications.

We don't know the cause of the issue, but I do know the result

We have env set on our web server set so that Perl is /our/own/private/bin/perl and not /usr/bin/perl, because this runs in a highly-networked and highly-clustered environment, mostly RedHat 6.8 and with 5.10.1 as system perl, if we want to have a consistent version and consistent modules, we need our own. This allows us to have #!/usr/bin/env perl as our shbang.

And this morning, for reasons I don't know, it stopped working. Whatever perl was being called, it wasn't /our/own/private/bin/perl. And this broke things.

One of the things that broke is this: Whatever perl is /usr/bin/env perl, it doesn't have Modern::Perl.

I'm for Modern Perl. My personal take is that chromatic and Modern Perl kept Perl alive in with Perl 5 while Larry Wall and the other language developers worked on Perl 6. Thus, I am grateful that it exists. But, while I was playing with it, I found a problem: Modern::Perl is not in Core, so you cannot rely on it being there, so a script might be running with a version greater than 5.8.8 and be able to give you everything you need, which to me is normally use strict, use warnings and use feature qw{say}, but if you're asking for Modern::Perl for it, it fails, and because you don't know which Modern thing you want, you don't know how to fix it.

This is part of my persistent hatred of magic. If it works and you don't understand how, you can't fix it if it stops working. I got to the heavy magic parts of Ruby and Rails and that, as well as "Life Happens", are why I stopped playing with it. And, I think, this is a contributing factor with this morning's brokenness.

2016/08/29

Perl and cpanm on Bash on Ubuntu on Windows 10: Who do I report this to?

I promised a longer post on Bash on Ubuntu on Windows 10, explaining why I use it, when I don't and why I even use Windows 10, but that isn't today.

Today is me trying to figure out who to report a bug to, and how to effectively report it.

The system (called "Ubuntu on Windows" for the rest of this blog post) is Ubuntu 14.04 LTS running kinda with Windows as the kernel instead of Linux. A good thing is that, if you can apt-get install a package, you can put it on Ubuntu on Windows. I don't think you can use it if 1) it connects to the kernel (because the kernel is Windows) or 2) it pops up a window. I've heard about people using XMing to get around that, but I haven't tried it yet.

The problem is that you get what you have available with dpkg, and the perl that comes with 14.04 is 5.18, where current is 5.24. Some modules haven't been packaged, and there's usually a gap between the package version and the cpan version, so I tend to use perlbrew and use my own perl instead of system perl.

And cpan and cpanm can't build on Ubuntu on Windows. I demonstrated this by trying to install YAML::XS with cpanm. I won't include the build log, but I will link to the build log. The problem comes at lines 146, 162 and 207:
#   Failed test 'Literal Scalar'
#   at t/dump-heuristics.t line 4.
#          got: '--- '1234567890
# 
#   1234567890
# 
#   1234567890
# 
# '
# '
#     expected: '--- |
#   1234567890
#   1234567890
#   1234567890
# '

#   Failed test 'Double Quoted Scalar'
#   at t/dump-heuristics.t line 10.
#          got: '--- 'A
# 
#   B
# 
#   C
# 
# '
# '
#     expected: '--- "A\nB\nC\n"
# '
# Looks like you failed 2 tests of 2.
t/dump-heuristics.t ...... 

...

#   Failed test 'Dumping Chinese hash works'
#   at t/utf8.t line 31.
#          got: '---
# Email: boss@opcafe.net
# 地址: 新竹市 300 石坊街 37-8 號
# 店名: OpCafé
# 時間: 11:01~23:59
# 電話: '03-5277806
# 
#   0991-100087
# 
# '
# '
#     expected: '---
# Email: boss@opcafe.net
# 地址: 新竹市 300 石坊街 37-8 號
# 店名: OpCafé
# 時間: 11:01~23:59
# 電話: "03-5277806\n0991-100087\n"
# '
# Looks like you failed 1 test of 8.
t/utf8.t ................. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/8 subtests 

The problem involves incorrectly handling newlines, but all this works and YAML::XS installs perfectly on my personal Perl, which I demonstrate below.
$ which perl ; perl -v | head -n 5 ; perl -MYAML::XS -e "print 'Yet Another Perl Hacker'"

/home/jacoby/perl5/perlbrew/perls/perl-5.24.0/bin/perl

This is perl 5, version 24, subversion 0 (v5.24.0) built for i686-linux
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2016, Larry Wall
Yet Another Perl Hacker

So, if it's Perl on Ubuntu on Windows, and the Perl works perfectly other places (including Strawberry Perl on the same Windows 10 machine (but only in PowerShell) ) and the Ubuntu works perfectly on other places, it must be the Windows, right?

That's what I thought, too. There's a GitHub repo for Ubuntu on Windows. Well, really, it's an issue tracker for Ubuntu on Windows, because there's nothing else in there. You can't clone and build your own. But, it does put the issue tracker in a place where the natural audience for Ubuntu on Windows would go. I put in an issue, but I think I might have asked wrong. I have cleaned up my issue some, but I see no sign that anyone else has seen it.

So, what is my next step? It isn't just YAML::XS; that's just a module I wanted at the time. It isn't just cpanm; I tried with cpan, too. It is just with Windows 10. Thoughts and suggestions from the greater Perl community?

2015/07/27

Things I learned for perlbrew: Config

Config.

Mostly, I haven't developed for Perl, I've developed for the perl on this machine. Sometimes, my perl on this machine, with this set of modules.

With perlbrew, you're starting with whatever Perl is available, and sometimes, upgrading the perl you're using. So, it's good to know something about that perl, isn't it?

So, how do we do that?

Config.

Look in there and you get api_revision, api_version and api_subversion, which allows you to know which version you are running.

Which makes me think that there are options here, if you're deploying software to places where they're not using the most recent perl.

In Javascript, they have a concept of polyfills, so that, if your page is loaded on an older browser with no HTML5 support, you can install a module that gives your old browser the capabilities it needs to do that.

Now, honestly, that seems a nicer way to handle things than
use 5.14 ; # Here's a nickel, buy yourself a Modern Perl
Doesn't it?

# pseudocode ahead. I'm just thinking through this
# polyfill_say.pl
use Config ;

use lib '$HOME/lib/Tools' ;

BEGIN {

if ( $Config{ 'api_revision' } < 5 || 
     $Config{ 'api_revision' } == 5 && $Config{ 'api_version' } < 10 ) {
        require Tools::JankySayReplacement qw{ say };
    }
}

say 'OK' ;

Of course there's perlbrew, plenv and just putting a modern perl in /opt/bin/perl or ~/bin/perl and being done with it. Just because I'm jazzed by an idea, that doesn't mean it's a good idea. But aren't a lot of the new cool things in Perl 5 just polyfilled back from Perl 6 these days? Shouldn't we be as nice to those stuck in 5.old as the Perl 6 people are to us?

Anyway, Config. It is in Core and it is cool.

2017/05/22

Testing Perl Modules on Windows: A Cry For Help

I've been working on modules again, after a recent push, and I found a big project whose .travis.yml file only went to Perl 5.20. I thought I'd get some dev brownie points by adding the next two stable versions, and found that my build was dying in the Pull Request submission process.

Specifically, it was dying on the checks. It passes Travis-CI, which runs the tests on Unix-like systems, but it was failing with Appveyor.

"What is Appveyor?", I asked myself.

Perhaps this isn't a direct quote.

Appveyor is a continuous integration service that tests against Windows systems. Scott Hanselman wrote a glowing review of it three years ago.

But there's no .appveyor.yml file in the project, so it runs and fails.

I've mentioned this on the project IRC, and no, I'm not going to name names, because there is movement toward testing on Windows, and even if it doesn't work, I admire the goal.

I wrote this three years ago, in response to a Python conference video:
2) Sure, real programmers use Unix/Linux to run their code, real programmers, but beginner programmers don't come in knowing how to set up an environment and come in with the (Windows) computer they have, and the documentation sucks and they feel lost and they don't like it and they don't feel the power, and they're gone. Even if you dislike Microsoft, good tools for Windows and good documentation are important for new programmers, important for building community, important for inclusiveness.
I run Windows, but I program on and for Linux, and only have one project based on Windows. But I have installed ActiveState and Strawberry Perls, and think that if you write general-purpose modules, you should be sure to test against Windows as well as Linux.

But, Travis-CI has documentation covering Perl projects. Appveyor says you can use Perl 5.20. eserte wrote a post on Appveyor for blogs.perl.org last year, but I'd love to see better documentation from either them, the Perl community or both. Following is the YAML from eserte, with a switch to "check only master branch", but as with Travis, which uses perlbrew and allows testing as far back as 5.8.8, I think having it test against older versions of Perl, both ActivePerl and Strawberry, would be the thing.

branches:
  only:
    - master

skip_tags: true

cache:
  - C:\strawberry



install:

  - if not exist "C:\strawberry" cinst strawberryperl

  - set PATH=C:\strawberry\perl\bin;C:\strawberry\perl\site\bin;C:\strawberry\c\bin;%PATH%

  - cd C:\projects\%APPVEYOR_PROJECT_NAME%

  - cpanm --installdeps .



build_script:

  - perl Makefile.PL

  - dmake test


If you have a more fully-featured .appveyor.yml file you'd like the Perl community to use, especially distinguishing MakeMaker modules from Dist::Zilla modules, I'd love to see it.

2017/07/06

Working Through Limitations: The Perl Conference 2017 Day 3

A Delorean with a Perl-powered center column and the cutest little Flux Capacitor on the dashboard. Oh, the wonders you can see at a developer conference.
"A man's got to know his limitations."

That's a line from Dirty Harry Callahan in Magnum Force, but it really described my planning for the Perl Conference. Once the calendar was up, I went in, first and foremost thinking "What are skills I need to learn?"

One crucial skill is version control. It's difficult to add to my main workflow, as I develop in production. (I live in fear.) But I'm increasingly adding it to my side projects. It is especially part of the process for maintaining the site for Purdue Perl Mongers, as well as aspects of HackLafayette, but beyond certain basics, I just didn't know much about how to use Git and version control to improve my projects. I learned how CPAN Testers tests your code on many platforms after you upload to CPAN, and how Travis-CL and Appveyor test against Linux, macOS and Windows after pushing to GitHub, but how track changes, align issues with branches, etc., are all new to me. So, I started Wednesday with Genehack and Logs Are Magic: Why Git Workflows and Commit Structure Should Matter To You. (Slides) I fully expect to crib ideas and aliases from this talk for some time to come.



There was a talk at a local software group featuring a project leader from Amazon on the technology involved with Alexa, which involved a lot of how this works, going from speech-to-text to tokenization and identifying of crucial keywords -- "Alexa, have Domino's order me a Pizza" ultimately boiling down to "Domino's" and "Pizza" -- and proceeding from there. It gave a sense of how Amazon is taking several hard problems and turning them into consumer tools.

What came very late in the talk is how to interface my systems with Amazon's "Talking Donkey", and I had a few conversations where we talked about starting the day with "Alexa, what's fresh hell is this?" and getting back a list of systems that broke overnight, but I lacked a strong idea of what is needed to make my systems interact with the Alexa system.

And an Echo.

But, thankfully, Jason Terry's Amazon, Alexa and Perl talk covered this, albeit more in the "Turn my smart lights on" sense than in the "Tell me what my work systems are doing" sense. Still, very much something I had been interested in.



But, as I implied, Amazon does a lot of heavy lifting with Alexa, getting it down to GETs and and POSTs against your API. If you're running this from home, where you have a consistent wired connection, this works. But, if you're running this, for example, in your car, you need it to be small, easy, and self-contained. Chris Prather decided to MAKE New Friends with a Raspberry Pi 3. This was a case of Conference-Driven Development, and he didn't have it ready to demonstrate at this time.



I've been trying to move my lab to the New Hotness over time, and because I will have to tie things back to existing systems, I have avoided learning too much in order to make it work. Joel Berger presented Vue.js, Mojolicious, and PostgreSQL chat in less than 50 lines. I've heard good things about Vue.js, which I heard from the project head was the parts he needed from Angular without the stuff he didn't use. (RFC Podcast) I use jQuery and Template and not much more, so the "only what you need" aspect sounds good to me. I fully expect to bug Joel on Twitter and IRC about this and other things I need to do with Mojolicious  over the coming months. (Slides)



But, of course, not every talk needs to speak directly to my short-term needs. Stevan Little has been trying to add an object system to the Perl 5 for quite some time, and in Hold My Beer and Watch This, he talks about Moxie, his latest attempt.



OK, the Moxie talk was in the same room as the Mojolicious talk and this next one. I could have gone to one of the others, but I decided against. Oh well. I would put "do more object-oriented development" as a thing to learn, so I was glad to hear it.

The last full talk was The Variable Crimes We Commit Against JavaScript by Julka Grodel. I would say that I code Javascript half as much as I code Perl, but twice as much as I code in other languages. I knew certain parts, like how let gives lexical scoping like Perl's my. I had heard about let from my JS Twitter list, as well as from Matt Trout's talk, but there were certainly things I didn't know.

And, actually, things I still don't. I had technical difficulties with my laptop, and if I could've worked it out that day, I would've tried to set up a lightning talk. Alas, not only did it not work -- in the end, I swapped out the WiFi, and I think if I switched back, it'd be sane again -- I missed a lot of her talk about arrow functions, which piqued the interest of Perl's functional programming guru, Mark Jason Dominus. (In a not-surprising addition to my limitations, I still haven't gotten to the end of Higher Order Perl.) Anyway, I believe that, after I finish this post, I will go back and watch this again.



After this, there were Lightning Talks, interspersed with ads for various Perl Mongers groups, and urges for you to start a Perl Mongers group. I am likely going to do another post, with these talks from all three days, but to end this one, I'll show off one where M. Allan Noah talks about the SANE Project and how he reverse engineers scanner output to allow him to support scanners without vendor input.



Lack of documentation is a limitation, but knowing the limitation does not mean you have to stop, and lack of documentation will not stop him. We should all draw inspiration from that.

Now that I'm two weeks past the end, most of the talks are up, and I would like to commend the organizers. Conference organization is hard, and this venue made it harder. The US Patent and Trademark Office is a great place, but there are aspects that they as a venue wanted secured and I would've preferred to be more open, but it was a beautiful venue and I be glad to return.

But, the thing I'd like to commend them most on is the high quality and fast turnaround on talk videos. The audio is good, the video is clear and well-lit, and the slides take precedence over the speaker in framing. It's everything I want in a conference video.

2015/08/17

I Am Unlikely To Move To Perl6

A camel in Busch Gardens in St. Louis.
I'm working on a project, which I will blog otherwise. This project is my hope to get into CPAN and level up my Perl game. I am still in progress, so it sits in the earliest stages of construction, with the bare minimum calling dzil new Foo::Bar will give you.

A friend asked me to rewrite it in Perl6.

He has also started putting out challenges, saying "Do this task in Perl5 without modules and I'll do it with Perl6.

I have several problems here. This project started with code I wrote to do something for work, and first and foremost, it had to do the work thing. That one part of the code could be useful to others is a perk. We're a Perl5 shop, so spending more than a few moments trying to do this task in Perl6 is just as counterproductive as trying to do it in Python, in Haskell, in Visual Basic or in x86 assembly. The library doesn't do everything, but it does everything I need, and the move from being a Perl developer to a CPAN developer is steep enough, thanks.

As for the challenges, I find them fundamentally unjust. If you have language A and language B, where feature X is available built into A but exists as a module in B, a module that comes with the core distribution of B, and this might not be a thing you use in most programs, then, to me, A puts unnecessary code into memory, while B is respectful of your requirements by not including things you don't need. My friend thinks the inclusion of feature X into A is a thing to be celebrated, a selling point he's trying to use to build excitement, while I regard it as a stacked deck.

But even as I write that, and intellectually agree with the logic I created, it isn't my fundamental objection. I could probably make a reverse argument if the tables were turned, and in the age where we have multiple gigabytes of RAM, and machines I have access to have a quarter-terabyte available, being so parsimonious with memory is out of style. As the saying goes, you cannot reason a person out of a position they didn't reason themselves into.

I started out as a Computer Science student, getting a second degree because my Journalism degree gave me no job. My school taught C as a subset of C++, using gcc on Solaris systems. This involved the use of stdio.h, and strings as arrays of characters. Intellectually, I get that it still is, but as a practical matter, it was not easy and it was not fun.

I worked helping to maintain the websites for a part of the school, and at first wrote HTML and documents in HTML, and also maintained the documentation library for the system administrators. Physical library. This is where I learned to love O'Reilly and Amazon. And, as I learned to program by learning CGI by learning Perl, where I learned that dealing with strings as scalars instead of arrays pointers to arrays of characters was a lot easier. This very much was where I learned to love Perl5.

This was an age of underpowered computers, and the computers I worked with were, in general, more underpowered than most. So weak, actually, that they could not run a web browser, which makes working as a web developer difficult. This is why I learned about X redirection. And, the minute I added any Javascript code to do anything on a page, this meant that the pages became unusable on the (for the time) beefy servers I was running the browsers on. I had reasoned myself into the position "pages + Javascript = slow and unusable".

Two things changed. The first was Time + Moore's Law = better computers. The second thing was the discovery of XMLHttpRequest, aka XHR or AJAX. This meant that Javascript was capable of more than display hacks, that you could do actual things with it. So, I grew to like Javascript, almost but not quite to the extent I like Perl.

A few years later, I had graduated and been hired by a local medical clinic. This was more IT than development, and I should've left early than I did, but while I was there, many of my friends had started to move to Python.

There was a computer lab where I had spent a great amount of time, and someone I knew had code that would identify who was at which SPARC5 machine, and which machines were empty, so you could know if it was worth your time to even go to that part of campus. I didn't need that information anymore, but what I needed was working Python code so I could learn how the language worked.

However, the program was indented with tabs. Python, for those unfamiliar with the Dynamic Language Wars, is a language where control structures are determined by indentation, instead of brackets, like most C-related languages. "Significant White Space" is the Ypres of the war between Perl and Python, the point where neither side will give ground.

The problem with tabs is that they tend to hide spaces. Which is to say, unless you have your editor set to show you the difference, SpaceTab looks exactly like Tab. This was exactly the problem that I found myself having. I was expecting, even wanting, to learn how if statements and for loops look in Python, and instead I found myself having to learn how Python error messages work.

The argument for white space is that, if you allow code to look like a one-line hairball, coders will make it look like a one-line hairball, but if your language design enforces coding standards, you will have readable code. I like reading readable code, and, as I said, I came from a Journalism background where I learned the rules to make printed documents more readable. It should be easy to reason me into appreciating Python, but having spent years learning Perl made me defensive, and spending hours struggling with with code entrenched my disgust. I viscerally dislike Python. I use it; my FitBit code is written in it because I found working through OAuth in Perl to be difficult. I'm willing to suck it up when necessary. But it is not my first choice of languages, and likely will never be, for entirely emotional reasons.

When I first started hearing about Perl6, I liked the idea of it. I wanted it to succeed. Perl is complex, they said. Only Perl can parse Perl, which isn't good. To hack Perl, to improve the base language, you need to know a bunch of domain-specific hackery, which means that the number of people who can make meaningful changes to the language is small and getting smaller. If we back up and start again, we can remove a bunch of cruft. This is fine.

And, since we're starting from scratch, we'll identify things we want, things that'll make the language better, and build those in. Logically, this works.

The brains of the Perl community went to work on Perl6, on the various platforms they tried to build it on. I can't create a strong argument against any individual technical decision. Use a VM to make it portable across many languages? Sure. Make our own VM system so we aren't reliant on someone else with interests unaligned with our own? Yeah, that's probably smart. But taken as a whole, that's a lot of things you need to do, with a significant chance of failure. The charitable might take it as a gamble worth taking, as a thing worth trying, even if it doesn't work right.

Not everyone is that charitable.

Meanwhile, those of us trying to use the language to do things to do things were left with, basically, a dead language. That is, until a bunch of us started to say "No!" We decided that, as we can, the good things about Perl6 will be taken and backported to Perl5, and that, while the world may identify the moment Jon Orwant's coffee mug hit the wall as the last moment they gave a shit about Perl, we have an enormous collection of pre-existing wheels called CPAN, which is considered the standard by which other languages' library systems are judged and found wanting.

So, to me, Perl5 is the language that saved me from C-type strings, the language that opened up computing for me and that has provided the majority of my paychecks for well over a decade. Perl6 is the language that has made me feel like a laughingstock for a good portion of that time. Perl5 is the a language that supported me, and Perl6 is the language that betrayed me.

Intellectually, I see that Perl6 has a real release date, that we're one community, that Perl6 isn't just a skunkworks for features to pull into Perl5. But I do not foresee my current workplace moving from Perl5 to Perl6, and I do not expect most place I might get hired into to be enthusiastic in embracing the change either.

I'm willing to change my mind. This is me trying to work through my issues enough to be open to Perl6. But it remains to be seen if I can change my heart.

2010/08/24

Overdue Book Review: Higher Order Perl

Ever promise to do something and not remember it?

That's kinda how it is here. I thought I was being given a copy of Higher Order Perl, while not understanding or promptly forgetting that I had incurred the responsibility of writing a review. Which is now way late, after far better reviews have already been writtten, and now that you can freely download it from the author's website.

I'll start by giving you the takeaway: Right now, there are two books that a Perl programmer should have and should read and should integrate into their coding style. The first is Perl Best Practices by Damian Conway, and this is there to raise the minimum level of your programs. If you want your code to be readable and reusable, read this book and get it to this level. The other is Higher Order Perl. This is because, while you may be leveraging the Awesome Power of Perl, you don't know the half of it. Mark Jason Dominus knows, and this book tells you.

I first learned about MJD when years ago I was going to defend Perl to my Linux User Group. One of my defense points was from MJD, who said of Norvig's seven features that make Lisp different, Perl shares six. Not that I really used it like that. I saw the bit about fibonacci sequences that shows up here in page 33, coded my own iterative and recursive implementations, became convinced that yes, I see the problem here, and wrote my version that used a hash or array to fix it. I didn't hit the Memoize part, which is where the awesome power comes in.

I knew there was awesome power there for a long time. I just failed to learn about it, because it was beyond what I needed at the time. Until, of course, it became part of what I needed. I needed it because I was creating a stack of horrible copy-and-paste code that really needed a dispatch table. I needed it because I needed a better and more adaptable filesystem spider than I had.

I am, for the first time, going chapter-by-chapter through Higher Order Perl and I like to think that my code will be better for it. I know two programs that's better for it.

2015/09/17

Logging, Plotting and Shoshin: A Developer's Journey

I heard about Log::Log4perl and decided that this would be a good thing to learn and to integrate into the lab's workflow.

We were having problems with our VMs and it was suggested I start logging performance metrics, so when we go to our support people, we have something more than "this sucks" and "everything's slow".

So, I had a reason and I had an excuse, so I started logging. But logs without graphs are boring. I mean, some logs can tell you "here, right here, is the line number where you are an idiot", but this log is just performance metrics, so if you don't get a graph, you're not seeing it.


That tells a story. That tells us that there was something goofy going on with genomics-test (worse, I can tell you, because we had nothing going on with genomics-test, because the software we want to test is not working yet. There was a kernel bug and a few other things that had fixed the other VMs, but not that one, so our admin took it down and started from scratch.

Look at that graph. Do you see the downtime? No?

That's the problem. This shows the last 100 records for each VM, but for hours where is no record, there should be -1 or a discontinuity or something.

I generally use R as a plotting library, because all the preformatting is something I know how to do in Perl, my language of choice, but I've been trying to do more and more in R, and I'm hitting the limits of my knowledge. My code, including commented-out bad trails, follows.


My thought was, hey, we have all the dates in the column final$datetime, and I can make a unique array of them. My next step would be to go through each entry for dates, and, if there was no genomicstest$datetime that equalled that date, I would throw in a null or a -1 or something. That's what the ifelse() stuff is all about.

But, I found, this removes the association between the datetime and the load average, and the plots I was getting were not the above with gaps, as it should be, but one where I'm still getting high loads today.

Clearly, I am looking at R as an experienced Perl programmer. You can write FORTRAN in any language, they say, but you cannot write Perl in R. The disjunct between a Perl coder codes Perl and an R coder codes R is significant. As a Perl person, I want to create a system that's repeatable and packaged, to get the data I know is there and put it into the form I want it to be. The lineage of Perl is from shell tools like sed and awk, but it has aspirations toward being a systems programming language.

R users are about the opposite, I think. R is usable as a scripting language but the general case is as an interactive environment. Like a data shell. You start munging and plotting data in order to discover what it tells you. In this case, I can tell you that I was expecting a general high (but not terribly high; we have had load averages into the hundreds from these VMs), and that the periodicity of the load came as a complete surprise to me.

(There are other other differences, of course. Perl thinks of everything as a scalar, meaning either a string or a number, or an array of scalars, or a special fast array of scalars. R thinks of everything as data structures and vectors and such. Things I need to integrate into my head, but not things I wish to blog on right now.)

The difference between making a tool to give you expected results and using a tool to identify unexpected aspects is the difference, I believe, between a computer programmer and a data scientist. And I'm finding the latter is more where I want to be.

So, I want to try to learn R as she is spoke, to allow myself to think in it's terms. Certainly there's a simple transform that gives me what I want, but I do not know it yet. As I do it, I will have to let go of my feelings of mastery of Perl and allow myself to become a beginner in R.

But seriously, if anyone has the answer, I'll take that, too.

2014/04/11

About Me


I've been working on my resume recently. No reason, really. (If you have questions, ask elsewhere. I'm fairly easy to find.) A recent response to my resume tells me it sucks at telling the story of me. Not too surprised; the form sucked before, and has become the gameboard for competative Reverse Buzzword Bingo, where if you find the right combination, you get the job, which makes it counterproductive to even try to tell your narrative.

So, this is the story of me:


My first degree was a BS in Journalism and Mass Communication. My greatest experience was as copy editor of my student paper, where I did full-page layout of the State-and-Nation wire news section to our paper's style standards, collected the weather forecast (which I cribbed from the regional newspaper's front page by calling a friend who worked in a gas station) and read every dang word in the paper. I got this position by copy-editing a copy of one week's front page, where I not only found the errors they introduced to test me, but also errors that made it into print that week. The biggest lessons I learned were that I was really good at finding miniscule errors in large blocks of text, I was pretty good and making an attractive and readable layout if I knew what sort of look we were shooting for, and if you're really physically and mentally exhausted with more work left to do, fruit juice is better than soda.

Also, I learned that when people say "If everyone from Dan Rather to the guy who writes obit for the local fish-wrap were to die tomorrow, there would not be enough positions opened up to employ all the people who graduate with journalism degrees this year", that's not just a punchline, that's a suggestion that you change majors as soon as this conversation ends.

xkcd
After not finding work in journalism, I went back to school, where I received a BS in Computer Science, and worked for the university as a web developer. As with xkcd, my time with Perl has been much more useful in my post-collegiate career than the courses I took.

Also, the things I learned as a j-school student -- finding small errors in large text blocks, making text blocks readable and attractive, and handling exhaustion -- are incredibly useful for CS students and web geeks, too.

During that time, I found that what I loved to do was converting things to other things. If you sent this email address a message with this string in the subject and a URL in the body, it responded with that URL's web page as the body. If you went to a certain web page, you could send an anonymous email. I fell in love early with RSS and wrote many things that read it and many other things that wrote it. I wrote a thing for my home page that displayed one guitar if I had new mail waiting, and another guitar if I didn't. I once helped write a tool that scrolled RSS headlines across the top of a web page. My goal and hope was to be a developer who turned things into other things, preferably with web technologies on a Unix/Linux platform while contributing to Open Source projects, in an organization whose primary work is creating software.

A thing I didn't really learn is that, in the last year of your college career, your primary goal is finding something to do with your life after graduation. Preferably, something where you are paid. Successfully completing your courses and graduating is necessary but secondary. So, I found myself as a VMS admin and occasional Microsoft-focused web developer for a medical clinic. I often made things out of other things: I recall parsing the four-digit long-distance records by long-distance code and sending a list of calls to the user of record for each code. It was okay, but not in line with my desired workplace as above.

After that, I came to work at the university again, for a research lab. We do gene sequencing; I do full-stack web development. I did some of the stuff I like to do: I turned web forms into database updates. I turned database tables into web tables. I turned CSV into plots with R. I used Perl and Linux and Unix and eventually R. I coded  daily.

For a while, I worked through a temp service with a defense contractor. Mostly I maintained a lab. On occasion, I set up specialty PCs. I used a visual pre-processor that eventually made C code, which wasn't fun. The pay was somewhat better, but of the things in the goals-and-hopes bit above, I did nothing.

So, I came back to the lab again. I gained more responsibility, and continue to make things from other things. I turn database dumps into XML-filled ZIP files that our instruments use to handle the sequences we give them. I used Sikuli and Jython to script a Windows application to turn a bunch of XML files that held UUencoded BLOBs into searchable XML files. I turned database dumps into JSON, and turned JSON into web page tables and Javascript-created plots. I write libraries that make it easy turn things on various machines into messages on my Linux box, on my Windows box, on my phone.

My one-true-language is Perl. Most everything I write, I first try in Perl, and only go away from it if I can't make it work there. I code in JavaScript: Perl doesn't run on the web browser, so I have to. I'm trying Node.js, and am slowly finding my way toward usefulness. I'm writing more and more R, because it's the shortcut for making plots.

I dislike Python. On first hearing about it, I thought, "What's the point? It sits in the same niche as Perl, but isn't Perl. Why use it?" My first experience with it was some code which displayed available machines in an on-campus computer lab. It has been handed to me as "this works", but it was indented with tabs, and somewhere along the line, a space had been inserted before a tab, so it didn't work. I get readable code, and my journalism degree forced me to appreciate whitespace for readability, but here something I couldn't see ruined my day. My Sikuli project was written with Jython, the Java implementation of Python, because that's the only choice, and my FitBit Tools are in Python because I couldn't easily get OAuth working with Perl. I could see using it more, but I write Perl in a Perl shop right now.

The Greyjoys of the Iron Islands Do Not Sow; I Do Not Make Web Pages. I make tools. I make things that turn things into other things. This can be web, but it will be dynamic, either at the server side or the client side. If you want a web page -- something static, a billboard for the web -- you will be happier and I will be happier if someone not me made this page. For design, I can get you 50-75% of the way to an attractive and useful web tool, and probably more if given a standard to align to, but design is not where my strengths lie and are not where my interests lie.

This is me. This is the tools I like to use, the tasks I like to complete, the environment I like to be in. Right now, I am a developer, working with web technologies in a Unix/Linux environment, doing more but not enough to contribute to Open Source projects. I'm part of an organization that's (writ large) education and/or (smaller) assisting biological research. This means I'm not quite where I want to be, but better than I was for many years. So, I'm looking, but not looking too hard.

2013/12/09

Something That Makes Me Sad, But Probably Shouldn't


Saturday was a good and incredibly geeky day for me. I went to the Indy Quantified Self Meetup, held in one coworking space, spent much of the afternoon in Fry's, where I met someone and talked about a startup idea (planned meeting, not happenstance), then went to Indy Hackers' Holiday Social, held in another coworking space. I listened to tech podcasts all of the way down and most of the way back. There's geeky things I didn't do (converse with my good geek friend Patrick) and otherly-geeky things that aren't for this blog (went to two guitar stores).

It was a good day, but there's one thing that wasn't so good. Maybe.

Above is the entire technical book stock at Fry's Indy. The entire shelf. There's lots and lots of shelves, and once, they were filled with all sorts of technical books. It fills my heart with joy to see a wall of O'Reillys. For the longest time, my first-pass rule for judging a technical project or tool was "Is there an O'Reilly book for it?" The books for learning Perl are Learning Perl and Programming Perl, and I judged Python harshly because the book a Python person would suggest is not Learning Python.

These days, though, your first pass for a technical question is asking Shub-Internet and ending up with a discussion thread, IRC log, or hopefully a Stack Overflow answer. I have editions 1 through 4 of Programming Perl, and they are decreasingly dogeared and beverage-stained as time goes on. I haven't really dug into Programming Perl, Fourth Edition, in part because I know how to write Perl, but in part because, even when I don't know what I'm doing, dead trees are not my go-to.

And, even from the perspective of the writer, writing tech books is a losing game. Jeff Atwood of Coding Horror writes about it when pushing a book, ASP.NET 2.0 Anthology, that he contributed to. xkcd did a comic (which I can't find) which compares the staying power of different book types, grouping math and physics texts as books that can stay without revision for decades, sometimes centuries, while a book on a language or protocol will be out of date very soon, sometimes before publication.

So, it would make sense to have Knuth's Art of Computer Programming on your shelf, but less so Perl, much less Perl 5.18.1. So, it would make sense to publish Art of Computer Programming on paper, but to generate and regenerate language documentation into electronic form.

And really, at places like Fry's, you want as much floor space devoted to things that can only be atoms, like motherboards, cases, LEDs, cables, trackpads and MacBook Pros, and things that can be bits should be bits. Plus, of course, the fact that you can put a whole lot of bits into your tablets and newsreader.

So, I can rationalize it at length, but it still saddens me to know that the world I used to live in, where there were places where a wall of amazement and possibility was available to me. I guess it outs me as a geezery geezer who geezes.

2017/06/19

Feeling "Tipsy": The Perl Conference 2017 Day 1

A few lessons learned before I dive into the talks I attended and what I gained from them:

  • I had figured that aspects of my Every Day Carry would not be appreciated in a government building, so I left my multitool off my belt, but, even after I had put everything I could think of into the tray, I had still made the machines go beep, causing the guards to pull out the wand. Tomorrow, pack lighter.
  • I take notes on a paper notebook, but I used my phone and tablet to connect to others via IRC and Twitter, as well as keeping track of my calendar, and I ran through my batteries before the end of the talks. In part, it was switching between the local WiFi and the cellular network, doing well with neither, but I'm not convinced it'd wouldn't be drained regardless. At lunch, I need to find a place to charge. I often come with a power strip for just this purpose, too.
  • I didn't break out the laptop once. If I don't use it more, I should just leave it and have less to carry.
Hopefully, I will remember this and come prepared for the next one.

Now, to the talks:

I started the day with MetaCPAN, the Grand Tour. I had been convinced earlier that MetaCPAN is better than CPAN for looking into modules, but there's more, including modules I am going to have start using to search CPAN.  

Graham Knop's Continuous Integration for CPAN followed. I had been aware of Travis-CI, and had become aware of AppVeyor recently, but the tooling available in Perl to work with these was less familiar to me. I was unaware that you can specify Linux or OSX in Travis, This was something I was thinking and asking questions of other developers about. I have issues on FreeBSD, which I'm told is something that Gitlab-CI can help me with, but somehow, I doubt I can connect Github to Gitlab, but I could be wrong.

Steven Lembark had much more with Dockerizing CPAN Testers: Running an Isolated Test Site than I could fit into my head, and I think I'll have to go back to the tape once it's available, but I think it's a useful addition to the world.

After lunch, I went to Joel Berger's Variables, Scoping and Namespaces, which he set as a talk for beginners. He went so far as to suggest more established developers go elsewhere. Since I never thought I learned all of Perl when I was learning, it was very much a lot of things I did already know, but a little "So that's why we do that", some more "Oooh. I forgot about that", and one weird trick that explains how to mock functions for tests.

(That, fundamentally, is my big item to work on as a developer. Testing.)

After this, I attended Matt S. Trout's ES6: Almost an acceptable Perl 5? and it gave me a sense of treating Javascript like Perl, but since I don't code Perl like Matt does, I probably won't code ES6 like Matt does. My notes peter out about halfway through, but they do include several gems, such as lodash, that might improve the code I do write.

Following this is Lightning Talks, which has a bunch of interesting points, going from "Write for OpenSource.com" to "Try learning Dart" to "Creating Test JSON from JSON Schemas" to "Using Instapaper, IFTTT and Buffer to tweet what you read" to the Alien modules, which I almost understand now. Or maybe not. Certainly, though, I'd be installing and trying Dave Rolsky's RTx-toGitHub right now if I wasn't so tired.

Finally, Sawyer X talked about Perl 5.26 and the changes that came and the changes that are coming. The thing that comes to mind first is that things that have been deprecated since 5.0 are finally being pulled. I understand the critics who think that removing . from @INC is insignificant, but I am still for it. I also like that Perl recognizes unhandled merges and dies now. 

Tomorrow, I will be learning about Dancer, Test2 and more with @INC, and visiting with family after

2016/04/12

Did I Mention I Hate Default Mail Notifications?

We live in a world of spam, of free email accounts and large mailing lists. You do not want to enable promiscuous notifications in such a world. That way lies madness.

But never knowing that the important people in your life — those you love, those who pay you, those who fix your guitars — are trying to contact you because you've turned off notifications is madness also. Perhaps a worse madness.

But Perl exists. CPAN exists. There is a way out.

I wrote a program for more general-purpose mail-handling, mostly clearing spam out of my work accounts, but decided to rewrite in order to handle the act of warning that I had new mail.

#!/home/jacoby/perl5/perlbrew/perls/perl-5.20.2/bin/perl

# specialized version of imap_task that handles just warnings. 
# problem with previous attempts is that it kept warning about
# new mail that matched until it was marked it read or deleted

# the goal is to do things once, with a data store independent 
# from IMAP that indicates if the warning has been sent. 

# YAML? JSON? Mongo? We'll try YAML.

use feature qw'say state' ;
use strict ;
use warnings ;
use utf8 ;

use Carp ;
use DateTime ;
use DateTime::Duration ;
use DateTime::Format::DateParse ;
use Getopt::Long ;
use IO::Interactive qw{interactive} ;
use IO::Socket::SSL ;
use Mail::IMAPClient ;
use YAML::XS qw{ LoadFile DumpFile } ;

use lib '/home/jacoby/lib' ;
use Locked ;
use Notify qw{ notify } ;
use Pushover ;
use Say qw{ say_message } ;

I have gone to perlbrew for most of my usable Perl, and I would normally use #!/usr/bin/env perl as my hashbang, but it is hard to tell crontab to use a perl other than system perl, so rather than trying, I specify the perl I want. Your usage will vary.

The next four non-code lines are my standard. That's mostly use Modern::Perl, I think, but I like being able to specify.

After that, there's a bunch of modules from CPAN. IO::Socket::SSL and Mail::IMAPClient are crucial for interacting with the mail server, YAML::XS is the better YAML module, according to Gabor Szabo. I like programs that give me verbose output when I run them, but don't clog my cron inbox when run via crontab, so I really overuse IO::Interactive. I am not sure that I need all the DateTime stuff I load for this purpose, but better safe than sorry.

Then there's the stuff that I wrote for purposes such as this. I have many programs that I want to behave differently if the computer is locked, which means I'm not at my standing desk, so I wrote Locked. I wanted to use notify-send on my Ubuntu machines to pop up notifications, so I wrote Notify. Net::Pushover wasn't written when I started this, so I wrote Pushover to interact with Pushover and should've put it on CPAN myself. Alas. And Say doesn't have to do with say(), but rather is a wrapper around eSpeak, a speech synthesizer.

my @sender ;
my $debug = 0 ;
my $task ;
$task = 'work_alert' ;

GetOptions(
    'debug=i' => \$debug,
    # 'task=s'  => \$task,
    )
    or exit(1) ;
# get the configuration
my $config_file = $ENV{HOME} . '/.imap/' . $task . '.yml' ;
croak 'No task set'  if length $task < 1 ;
croak 'No task file' if !-f $config_file ;

my $settings = LoadFile($config_file) ;
$settings->{debug} = $debug ;

# set a message if one hasn't been set
$settings->{message} = $settings->{message} ? $settings->{message} : 'You have mail' ;

my $has_spoken = 0 ;

say {interactive} '='x20;
my $warn_file   = $ENV{HOME} . '/.imap_warn.yml' ;
my $warnings = LoadFile($warn_file) ;
check_imap($settings) ;
DumpFile( $warn_file , $warnings ) ;
say {interactive} '-'x20;
exit ;

Here I establish a bunch of globals and everything up for check_imap(), the main part of this program.

There are two YAML files that this program uses. One is .imap_warn.yml, which is a hash where the key is "$FROM||$SUBJECT||$DATE" and the value is 1, so I can tell if I've been told about a certain email before, and .imap/work_alert.yml, which is the main configuration file, and looks like this:

---
server: mailserver.example.com
port: 993
username: username
password: you_dont_get_my_password
message: 'You have mail'
folders:
    INBOX: 
        alert:
            subject:
                - 'big data'
            from:
                # Family
                - jacoby
                # The Lab
                - boss@example.com

I have used a separate file to hold the specifications for my SMTP and IMAP servers, but here, having all the config in one place seemed right. Since it contains password information, it is especially important that permissions are set correctly, specifically only you can read it. I do not test permissions in this program.

As mentioned, this is adapted from a more general mail-handling program, which takes specific configuration files for the kind of work it does. This just has the one, so that has been commented out, leaving just the debug flag.

I have had issues with YAML empty-writing files, which is why I separated .imap_warn from work_alert.pl.
sub check_imap {
    my $settings = shift ;
    my $client ;
    if ( $settings->{port} == 993 ) {

        my $socket = IO::Socket::SSL->new(
            PeerAddr => $settings->{server},
            PeerPort => $settings->{port},
            )
            or die "socket(): $@" ;

        $client = Mail::IMAPClient->new(
            Socket   => $socket,
            User     => $settings->{username},
            Password => $settings->{password},
            )
            or die "new(): $@" ;
        }
    elsif ( $settings->{port} == 587 ) {
        $client = Mail::IMAPClient->new(
            Server   => $settings->{server},
            User     => $settings->{username},
            Password => $settings->{password},
            )
            or die "new(): $@" ;
        }

    my $dispatch ;
    $dispatch->{'alert'}          = \&alert_and_store_mail ;
    $dispatch->{'warn'}           = \&warn_mail ;

    if ( $client->IsAuthenticated() ) {
        say {interactive} 'STARTING' ;

        for my $folder ( keys %{ $settings->{folders} } ) {
            say {interactive} join ' ', ( '+' x 5 ), $folder ;
            $client->select($folder)
                or die "Select '$folder' error: ",
                $client->LastError, "\n" ;

            my $actions = $settings->{folders}->{$folder} ;

            for my $msg ( reverse $client->unseen ) {
                my $from = $client->get_header( $msg, 'From' ) || '' ;
                my $to   = $client->get_header( $msg, 'To' )   || '' ;
                my $cc   = $client->get_header( $msg, 'Cc' )   || '' ;
                my $subject = $client->subject($msg) || '' ;

                say {interactive} 'F: ' . $from ;
                say {interactive} 'S: ' . $subject ;

                # say { interactive } 'T: ' . $to ;
                # say { interactive } 'C: ' . $cc ;

                for my $action ( keys %$actions ) {

                    # say { interactive } '     for action: ' . $action ;

                    for my $key ( @{ $actions->{$action}->{from} } ) {
                        if (   defined $key
                            && $from =~ m{$key}i
                            && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{to} } ) {
                        if ( $to =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{cc} } ) {
                        if ( $cc =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{subject} } ) {
                        my $match = $subject =~ m{$key}i ;
                        if ( $subject =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    }
                say {interactive} '' ;
                }

            say {interactive} join ' ', ( '-' x 5 ), $folder ;
            }

   # $client->close() is needed to make deletes delete, put putting before the
   # logout stops the process.
        $client->close ;
        $client->logout() ;
        say {interactive} 'Finishing' ;
        }
    say {interactive} 'Bye' ;
    }

There are four things we can match on: from, to, cc and subject. I generally match on subject and from, but the code is there.

I have started but not finished Higher Order Perl by Mark Jason Dominus, but one of the things I got from that book (I think; if not, then from co-workers) is the concept of a dispatch table, where behavior of the program changes based on the data. I could simplify this a lot more, I'm sure, with more higher-order programming, but I'm reasonably happy with it right now.

# ====================================================================
# send to STDOUT without IO::Interactive, for testing
sub warn_mail {
    my ( $client, $msg ) = @_ ;
    say {interactive} 'warn' ;
    my $from = $client->get_header( $msg, 'From' ) || return ;
    my $to   = $client->get_header( $msg, 'To' )   || return ;
    my $subject = $client->subject($msg) || return ;
    my $date  = $client->get_header( $msg, 'Date' ) || return ;
    my $dt    = DateTime::Format::DateParse->parse_datetime($date) ;
    my $today = DateTime->now() ;
    $dt->set_time_zone('UTC') ;
    $today->set_time_zone('UTC') ;
    my $delta = $today->delta_days($dt)->in_units('days') ;
    say $from ;
    say $to ;
    say $subject ;
    say $dt->ymd ;
    say $delta ;
    }

# ====================================================================
# alert about new mail
sub alert_and_store_mail {
    my ( $client, $msg ) = @_ ;
    say {interactive} 'alert and store' ;
    my $date = $client->get_header( $msg, 'Date' ) || 'NONE' ;
    my $from = $client->get_header( $msg, 'From' ) || 'NONE' ;
    my $to   = $client->get_header( $msg, 'To' )   || 'NONE' ;
    my $subject = $client->subject($msg) || 'NONE' ;
    my $key = join '||' , $from , $subject , $date ;
    $key =~ s{\s+}{ }g ;
    my $title =  'Mail From: ' . $from ;
    chomp $title ;
    chomp $subject ;

    return if $warnings->{$key} ;
    $warnings->{$key} = 1 ;

    $from =~ s{\"}{}gx ;
    if ( is_locked() ) {
        pushover(
            {   title   => $title ,
                message => $subject
                }
            ) ;
        }
    else {
        say {interactive} $title  ;
        say {interactive} $subject ;
        say {interactive} defined $warnings->{$key} ? 1 : 0 ;
        say {interactive} 'has spoken: ' . $has_spoken ;
        if ( ! $has_spoken ) {
            say_message( { message => $settings->{message} , title => '' } ) ;
            }
        notify(
            {   title   => $title ,
                message => $subject ,
                icon    => '/home/jacoby/Dropbox/Photos/Icons/mail.png' ,
                }
            ) ;
        }
    $has_spoken = 1 ;
    return ;
    }

warn() is useful for debugging, but the work of the program is done in alert_and_store_mail(). $client is the IMAP connection, and $msg is the message itself. I find that I have to send both. I might be doing it wrong, though.

And here is where my modules come in. is_locked() returns a boolean, depending on which way you lock your screens. say_message(), pushover() and notify() share a format, a hashref containing title and message. say_message() tells me that I have notifications coming, and they show up on my desktop. And if I'm away from my desk, they show up on my tablet because of Pushover.

I'll put this into a repo on GitHub, including all the modules. I would like to get this into shape to be something like App::imap_warn or the like, but I'm not there yet. I'm sure there's interest, because default notifications suck.

2017/06/29

Three Little Words: The Perl Conference 2017 Day 2

I lost the bubble.

I wrote a post at the end of the first day of the Perl Conference, intending to do the same for the second and third, but I went to visit family and play with their puppies on Tuesday night, and puppies are better than blogging.

Puppies!
The beginning of the third day, my laptop hit a weird issue where it couldn't enable the wireless NIC. I fixed it at home, once I had access to precision screwdrivers again, but this meant that I couldn't blog the last day, either.

But waiting affords me the opportunity to add videos of the talks I attended. So, yay.

I started Day 2 with Sam Batschelet's Dancing In The Cloud, which was not what I was expecting, being more a deep dive in another domain. I was hoping for an introduction to using the Dancer framework on cloud providers. It was interesting, and I now know what kubernetes does and that etcd exists, but this is far enough away from where I live that I don't know if it'll give me much more than buzzword recognition when I next attend a technical conference.


Again, I would like to reiterate that it was deep for me. Good talk, good speaker, but I was the wrong audience. Read the talk descriptions, not just the titles!

This was followed by Chad Granum on Test2, which is not yet on YouTube. I came to TPC wanting to up my testing game, and I've been looking through Andy Lester's modules and brian d foy's Stack Overflow posts trying to get into the mindset. This talk is a long-form walk through the diffs between the old and new test frameworks, and by showing how to test using Test2, it showed a lot about how to test in general. Once the talk is up, I am sure I will go through it again and again, trying to get my head around it.


But just because we don't have this talk on YouTube, that doesn't mean we don't have Chad talking about testing.

My next talk was Adventures in Failure: Error handling culture across languages by Andrew Grangaard, which is another point of growth for me. All too often, I just let errors fall, so knowing how they're commonly done is a good thing, well presented.


Damian Conway was the keynote speaker, and his talk was called Three Little Words. Those are, of course, I ❤ Perl.

Also, class has method. Conway proceeded to implement Perl6-style OOP with Dios, or Declarative Inside-Out Syntax, which needed Keyword::Declare to enable the creation of new keywords in Perl 5. To get this going, he needed to more quickly parse Perl, so along comes PPR, the Pattern-based Perl Recognizer. It was in this talk that the previous Lightning Talk from Chad, including his module Test2::Plugin::Source, comes from.

The Lightning Talks were good, but the one that comes up as a good one to put here is Len Budney's Manners for Perl Scripts. This makes me want to look into CLI::Startup.


I hope to get to Day 3 and maybe another post just on Lightning Talks soon. An hour's dive into topics won't get you too deep, but an hours' lightning talks will will open up a number of possibilities for you to follow.

2009/03/26

Setting your GTalk Status with Perl

When confronted with a problem, I used to go to Google and type perl problem-domain , but I stopped that, because that, at best, sent me to a module page on CPAN, which, unfortunately, doesn't always tell you how to use the module.

I have started to use perl problem-domain site:blogspot.com instead, as it is likely that someone hit the same problem and found a solution. When I started looking into the problem of changing the status for Google Talk, I did this, and found this page, ending with the line "Worked? Or Crap?" I could not help responding "Worked AND Crap."

"Worked" is nothing to sneeze at. The IETF motto is "rough consensus and running code", and when you have code that does what you ask it to, what more do you want? But when you use the same variable name three times for three different things, you have to wonder if you could do a better job with it.


#!/usr/bin/perl
# http://digitalpbk.blogspot.com/2009/02/perl-change-google-talk-status.html
#usage : perl stat.pl

use Modern::Perl ;
use Net::XMPP ;
use subs qw{ get_status set_status } ;

my $status = get_status ;
if ( defined $status && length $status > 1 ) {
set_status $status ;
}
exit ;

sub get_credentials {
#this is the point where my setup sucks
my %creds ;
$creds{ username } = 'NO' ;
$creds{ password } = 'A Thousand Times NO' ;
return %creds ;
}

sub get_status {
return join ' ', @ARGV ;
}

sub set_status {
my $status = shift ;
my %cred = get_credentials ;
my $username = $cred{ username } ;
my $password = $cred{ password } ;
my $hostname = 'talk.google.com' ;
my $port = 5222 ;
my $componentname = 'gmail.com' ;
my $connectiontype = 'tcpip' ;
my $tls = 1 ;
my $Con = new Net::XMPP::Client( debuglevel => 0 ) ;
my $con_status = $Con->Connect( hostname => $hostname,
port => $port,
componentname => $componentname,
connectiontype => $connectiontype,
tls => $tls,
timeout => 10 ) ;

if ( !( defined( $con_status ) ) ) {
exit( 0 ) ;
}
my $sid = $Con->{ SESSION }->{ id } ;
$Con->{ STREAM }->{ SIDS }->{ $sid }->{ hostname } = $componentname ;
my @result = $Con->AuthSend( username => $username,
password => $password,
resource => "neuron" ) ;
$Con->Send(
"<iq type='get' to='gmail.com'>"
) ;
$Con->Process() ;
my $iq = $Con->SendAndReceiveWithID(
"<iq type='get' to='$username\@gmail.com'>"
) ;
my ( $cur_status, $statuslist, $show ) = ( "", "", "" ) ;
$cur_status = $1 if ( $iq->GetXML() =~ m/(.*?)<\/status>/ ) ;
$statuslist = $1
if ( $iq->GetXML() =~ m/(<status-list(.*)<\/status-list>)/ ) ;
$show = $1 if ( $iq->GetXML() =~ m/<show>(.*?)<\/show>/ ) ;
##Change status
$Con->Send(
"<iq type='set' to='$username\@gmail.com'>
<query xmlns='google:shared-status'>
<status>$status

<show>$show
</query></iq>" ) ;
$Con->Process() ;
$Con->Disconnect() ;
return $status ;
}


I'll admit that set_status is fairly cargo-cult. But I think it is a little less crap that it was before. Thanks to digitalpbk for the code.

2009/10/20

Perl IS a community!

Ever have code that doesn't work and you don't know why? I do on occasion, and the error messages didn't tell me anything useful, so I decided to try Perl::Critic on it. This is less a tool to debug your non-functioning program than a program that tells you a better way to have your functioning program function, but I thought it could tell me something.

I generally start all my programs these days with:

use Modern::Perl ;
This takes the place of this:

use 5.010 ;
use strict ;
use warnings ;
This is very nice, in that it gets you to the cool stuff in Perl in one line.

But...

Perl::Critic wants you to run using strict and warnings, and when you use Modern::Perl, it doesn't explicitly say that you are using them, and Critic dinks you for it.

I thought that was interesting.

And, today, when you find something interesting, you tweet it. I also put it on Identi.ca, which is like twitter but mostly programmer types go there. And I got a response. From chromatic on identi.ca and from the Critic folks on Twitter.

That just warms my heart. Not specifically the help to make it work, which is of course great, but that the developers are listening.

And, incidently, making a .perlcritic file with this in it makes it work:

equivalent_modules = Modern::Perl



2013/07/30

Telling One Liners

I don't do a lot of Perl one-liners. In general, I think if you're writing Perl, you're going to want to repeat it so you should put it in a file.

I say in general, because I ran into a specific situation. For a plot-generating process, we 1) pointed to the version of the plot-maker that didn't use libcairo, so it couldn't be used without X11, and 2) accidentally removed lots from the completed file, where we list the plots we've done so we don't redo them, which lead to 3) remaking lots of plots that exist with a plot-maker that couldn't make plots, meaning files that existed got replaced with zero-sized files.

Once we figured out what the problem was, I wrote this one-liner, which ran for over an hour and 40 minutes.
for i in {1..100} ; 
do (
    ls -lS */nanodrop/*png | perl -lane ' print $F[8] if 0 == $F[4] ' | wc -l 
    ); 
sleep 60 ; 
done 
Basically, it looped over and over again to keep constant watch (the for loop), each time taking an ls -l (the -S, order by size, not strictly being necessary but left over from my initial checking), breaking it apart with Perl, printing the file name ($F[8]) if the file size ($F[4]) was 0. I pipe the output into wc -l,  which counts the number of lines. I then sleep for a minute (as the number of zero-length files was over 7000 at first run, the ls-l took quite some time, so the reports were much longer than a minute apart), and do it again.

Granted, I could've done it all in Perl, but why re-implement tools that exist in the shell when you don't have to? I could've done the Perl stuff in Awk, if only I knew Awk better, but I could get to the point where I knew the damage and the rate of repair much faster with tools I knew. Really, the only thing I needed to learn was that perl -lane was what I needed to get the lines broken easily.

2015/06/22

"Well, That Was Strange": Hunting Gremlins in SQL and Perl

The query base is 90 lines.

Depending on what it's used for, one specific entry or the whole lot, it has different endings, but the main body is 90 lines. There are 20 left joins in it.

It is an ugly thing.

So ugly, in fact, that am loath to include it here.

So ugly that I felt it necessary to comment and explain my use of joins.

This is where the trouble started.

I noticed it when I was running tests, getting the following error.


Clearly, it needed a bind variable, but something along the line blocked it.

I had this problem on Friday morning on our newer server, then it stopped. Honestly, it was such a fire-fighting day that I lost track of what was happening with it.

Then the module was put on the old server and the problem rose up again.

Whether that makes me Shatner or Lithgow
remains an exercise for the reader.
I said "my code has gremlins" and went home at 5pm.

When I got back to the lab this morning, I made different test scripts, each identical except for the hashbang. I set one for system Perl, which is 5.10, one for the one we hardcode into most of our web and cron uses, which is 5.16, and the one we have as env perl, currently 5.20.

The cooler solution would've been to have several versions of Perl installed with Perlbrew, then running perlbrew exec perl myprogram.pl instead, but I don't have Perlbrew installed on that system.

The error occurs with 5.10. It does not with 5.16 or 5.20.

And when I run it against a version without the comments in the query, it works everywhere.

I don't have any clue if the issue is with Perl 5.10 or with the version of DBI currently installed with 5.10, and I don't expect to. The old system is a Sun machine that was off support before I was hired in, and the admin for it reminds us each time we talk to him that it's only a matter of time before it falls and can no longer get up. I haven't worked off that machine for around two years, and this query's move to the old server is part of the move of certain services to the new machine.

And, as everything is fine with Perls 5.16 or higher, I must regard this as a solved problem except with legacy installs.

I know that MySQL accepts # as the comment character, but Sublime Text prefers to make -- mean SQL comments, so when I commented the query, I used the double-dash, and our solution is to remove the comments when deploying to the old server. It's a temporary solution, to be sure, but deploying to the old server is only temporary, too.

It's a sad and strange situation where the solution is to uncomment code, but here, that seems to be it.

Update: Matt S. Trout pushed me to check into the DBD::mysql versions, to see which versions corresponded to the error. The offending 5.10 perl used DBD::mysql v. 4.013, and looking at the DBD::mysql change log, I see bug #30033: Fixed handling of comments to allow comments that contain characters that might otherwise cause placeholder detection to not work properly. Matt suggests adding "use DBD::mysql 4.014;", which is more than reasonable.

2009/10/19

Modern One-Liner Perl

I have never been much of a fan of Perl one-liners. Take every slur about Perl that exists in the popular imagination, and that's the reality for one-liners.

perl -e 'print 51 * 51 , "\n"'
But now, by going -E instead of -e, you bring in Perl 5.10. This means you get say, which is by far my fave Perl 5.10 addition.

perl -E 'say 51 * 51'
I'm already starting to do one-liners more, now that I have say. Using realias is also helpful with this.

Credit where due.



2016/09/01

Perl on Ubuntu on Windows: Finding The Right Test Case



I'm still hung on getting CPAN working for Ubuntu on Windows. In the comments, Chas. Owens gave me great advice for proceeding with this problem:
Write a minimal test case, run the test case with strace on Ubuntu on Windows and Ubuntu on Linux. The outputs are supposed be identical, if they aren't (and we expect them not to be because of the error), then you have a good shot at identifying the misbehaving syscall (which is the only difference between Ubuntu on Windows and Ubuntu on Linux).

Once you see the difference, look into what the syscall does and try to determine which system is implementing it correctly (probably Linux). If it is Windows post to the github tracker with the test case and the identified syscall. If it is Linux, then report it to Ubuntu.
My first thought was to do this within cpanm, but I thought sudo su then strace -o platform -ff strace cpanm YAML::XS was a bit much. In part because when platform was Linux, it generated one file and hundreds on Windows.

Then it struck me that instead, I should just focus on the tests themselves. I cloned Ingy döt Net's YAML repo and tried to run prove test/ in both places. Went fine with Linux, failed with Windows. Butrealized after a second, it succeeded while using my personal perl, not system perl. /usr/bin/prove test/ failed on Ubuntu. apt-get install libtest-base-perl on both systems helped a lot, but now it wants Test::More, (I know because I searched for what modules the tests are asking for.)

For all I know, there's a package that provides Test::More, but it isn't libtest-more-perl, and digging further into that seems like a waste.

So I'm thinking it through again, looking at a failing test in YAML::XS:

use t::TestYAMLTests tests => 2;
use utf8;

is Dump("1234567890\n1234567890\n1234567890\n"), "--- |
  1234567890
  1234567890
  1234567890
", 'Literal Scalar';

is Dump("A\nB\nC\n"), q{--- "A\nB\nC\n"} . "\n", 'Double Quoted Scalar';


By "failing test" I am saying it works in natural Linux but not Ubuntu on Windows. And it's striking me: I need to find Dump. Where is Dump? In the C. It is an XS module, is it not? So, it's striking me that the solution is in C.

Which means I have to write C.

More later.

I think there's only been one time when I coded C on the clock, and only one time when my C knowledge was required on the clock.

The former was at a former workplace, where I wrote and compiled some C to compare UltraEdit with another editor, so I could help decide which we were going to buy a site license for. As I can only remember UltraEdit, I can say that's the one I liked better. The code itself was scarcely better than Hello World.

The latter was at another former workplace, where there was a tool that allowed mechanical engineers to drag together components like traces, and then first turned those traces into C code and compiled them. There was an issue where it wouldn't work, and I found the error logs and worked back.

I'm looking at perl_libyaml.c. I'm looking at perl_libyaml.h. I don't have nearly enough C skills to start here.
/*
 * This is the main Dump function.
 * Take zero or more Perl objects and return a YAML stream (as a string)
 */
void
Dump(SV *dummy, ...)
{
    dXSARGS;
    perl_yaml_dumper_t dumper;
    yaml_event_t event_stream_start;
    yaml_event_t event_stream_end;
    int i;
    SV *yaml = sv_2mortal(newSVpvn("", 0));
    sp = mark;

    set_dumper_options(&dumper);

    /* Set up the emitter object and begin emitting */
    yaml_emitter_initialize(&dumper.emitter);
    yaml_emitter_set_unicode(&dumper.emitter, 1);
    yaml_emitter_set_width(&dumper.emitter, 2);
    yaml_emitter_set_output(
        &dumper.emitter,
        &append_output,
        (void *) yaml
    );
    yaml_stream_start_event_initialize(
        &event_stream_start,
        YAML_UTF8_ENCODING
    );
    yaml_emitter_emit(&dumper.emitter, &event_stream_start);

    dumper.anchors = newHV();
    dumper.shadows = newHV();

    sv_2mortal((SV *)dumper.anchors);
    sv_2mortal((SV *)dumper.shadows);

    for (i = 0; i < items; i++) {
        dumper.anchor = 0;

        dump_prewalk(&dumper, ST(i));
        dump_document(&dumper, ST(i));

        hv_clear(dumper.anchors);
        hv_clear(dumper.shadows);
    }

    /* End emitting and destroy the emitter object */
    yaml_stream_end_event_initialize(&event_stream_end);
    yaml_emitter_emit(&dumper.emitter, &event_stream_end);
    yaml_emitter_delete(&dumper.emitter);

    /* Put the YAML stream scalar on the XS output stack */
    if (yaml) {
        SvUTF8_off(yaml);
        XPUSHs(yaml);
    }
    PUTBACK;
}

Hrm. Maybe back to the Perl.

Looking at YAML.pm, YAML::Dumper.pm and YAML::Dumper::Base.pm make it seem like you could make a master class on how modules go together. Previously, when I've dived in to figure out issues, yeah, there's lots of confusing parts but I could find my way to the broken thing. perl_libyaml.c is looking much more approachable now.