Cookie Notice

As far as I know, and as far as I remember, nothing in this page does anything with Cookies.
Showing posts with label cpan. Show all posts
Showing posts with label cpan. Show all posts

2016/09/01

Perl on Ubuntu on Windows: Finding The Right Test Case



I'm still hung on getting CPAN working for Ubuntu on Windows. In the comments, Chas. Owens gave me great advice for proceeding with this problem:
Write a minimal test case, run the test case with strace on Ubuntu on Windows and Ubuntu on Linux. The outputs are supposed be identical, if they aren't (and we expect them not to be because of the error), then you have a good shot at identifying the misbehaving syscall (which is the only difference between Ubuntu on Windows and Ubuntu on Linux).

Once you see the difference, look into what the syscall does and try to determine which system is implementing it correctly (probably Linux). If it is Windows post to the github tracker with the test case and the identified syscall. If it is Linux, then report it to Ubuntu.
My first thought was to do this within cpanm, but I thought sudo su then strace -o platform -ff strace cpanm YAML::XS was a bit much. In part because when platform was Linux, it generated one file and hundreds on Windows.

Then it struck me that instead, I should just focus on the tests themselves. I cloned Ingy döt Net's YAML repo and tried to run prove test/ in both places. Went fine with Linux, failed with Windows. Butrealized after a second, it succeeded while using my personal perl, not system perl. /usr/bin/prove test/ failed on Ubuntu. apt-get install libtest-base-perl on both systems helped a lot, but now it wants Test::More, (I know because I searched for what modules the tests are asking for.)

For all I know, there's a package that provides Test::More, but it isn't libtest-more-perl, and digging further into that seems like a waste.

So I'm thinking it through again, looking at a failing test in YAML::XS:

use t::TestYAMLTests tests => 2;
use utf8;

is Dump("1234567890\n1234567890\n1234567890\n"), "--- |
  1234567890
  1234567890
  1234567890
", 'Literal Scalar';

is Dump("A\nB\nC\n"), q{--- "A\nB\nC\n"} . "\n", 'Double Quoted Scalar';


By "failing test" I am saying it works in natural Linux but not Ubuntu on Windows. And it's striking me: I need to find Dump. Where is Dump? In the C. It is an XS module, is it not? So, it's striking me that the solution is in C.

Which means I have to write C.

More later.

I think there's only been one time when I coded C on the clock, and only one time when my C knowledge was required on the clock.

The former was at a former workplace, where I wrote and compiled some C to compare UltraEdit with another editor, so I could help decide which we were going to buy a site license for. As I can only remember UltraEdit, I can say that's the one I liked better. The code itself was scarcely better than Hello World.

The latter was at another former workplace, where there was a tool that allowed mechanical engineers to drag together components like traces, and then first turned those traces into C code and compiled them. There was an issue where it wouldn't work, and I found the error logs and worked back.

I'm looking at perl_libyaml.c. I'm looking at perl_libyaml.h. I don't have nearly enough C skills to start here.
/*
 * This is the main Dump function.
 * Take zero or more Perl objects and return a YAML stream (as a string)
 */
void
Dump(SV *dummy, ...)
{
    dXSARGS;
    perl_yaml_dumper_t dumper;
    yaml_event_t event_stream_start;
    yaml_event_t event_stream_end;
    int i;
    SV *yaml = sv_2mortal(newSVpvn("", 0));
    sp = mark;

    set_dumper_options(&dumper);

    /* Set up the emitter object and begin emitting */
    yaml_emitter_initialize(&dumper.emitter);
    yaml_emitter_set_unicode(&dumper.emitter, 1);
    yaml_emitter_set_width(&dumper.emitter, 2);
    yaml_emitter_set_output(
        &dumper.emitter,
        &append_output,
        (void *) yaml
    );
    yaml_stream_start_event_initialize(
        &event_stream_start,
        YAML_UTF8_ENCODING
    );
    yaml_emitter_emit(&dumper.emitter, &event_stream_start);

    dumper.anchors = newHV();
    dumper.shadows = newHV();

    sv_2mortal((SV *)dumper.anchors);
    sv_2mortal((SV *)dumper.shadows);

    for (i = 0; i < items; i++) {
        dumper.anchor = 0;

        dump_prewalk(&dumper, ST(i));
        dump_document(&dumper, ST(i));

        hv_clear(dumper.anchors);
        hv_clear(dumper.shadows);
    }

    /* End emitting and destroy the emitter object */
    yaml_stream_end_event_initialize(&event_stream_end);
    yaml_emitter_emit(&dumper.emitter, &event_stream_end);
    yaml_emitter_delete(&dumper.emitter);

    /* Put the YAML stream scalar on the XS output stack */
    if (yaml) {
        SvUTF8_off(yaml);
        XPUSHs(yaml);
    }
    PUTBACK;
}

Hrm. Maybe back to the Perl.

Looking at YAML.pm, YAML::Dumper.pm and YAML::Dumper::Base.pm make it seem like you could make a master class on how modules go together. Previously, when I've dived in to figure out issues, yeah, there's lots of confusing parts but I could find my way to the broken thing. perl_libyaml.c is looking much more approachable now.

2016/08/29

Perl and cpanm on Bash on Ubuntu on Windows 10: Who do I report this to?

I promised a longer post on Bash on Ubuntu on Windows 10, explaining why I use it, when I don't and why I even use Windows 10, but that isn't today.

Today is me trying to figure out who to report a bug to, and how to effectively report it.

The system (called "Ubuntu on Windows" for the rest of this blog post) is Ubuntu 14.04 LTS running kinda with Windows as the kernel instead of Linux. A good thing is that, if you can apt-get install a package, you can put it on Ubuntu on Windows. I don't think you can use it if 1) it connects to the kernel (because the kernel is Windows) or 2) it pops up a window. I've heard about people using XMing to get around that, but I haven't tried it yet.

The problem is that you get what you have available with dpkg, and the perl that comes with 14.04 is 5.18, where current is 5.24. Some modules haven't been packaged, and there's usually a gap between the package version and the cpan version, so I tend to use perlbrew and use my own perl instead of system perl.

And cpan and cpanm can't build on Ubuntu on Windows. I demonstrated this by trying to install YAML::XS with cpanm. I won't include the build log, but I will link to the build log. The problem comes at lines 146, 162 and 207:
#   Failed test 'Literal Scalar'
#   at t/dump-heuristics.t line 4.
#          got: '--- '1234567890
# 
#   1234567890
# 
#   1234567890
# 
# '
# '
#     expected: '--- |
#   1234567890
#   1234567890
#   1234567890
# '

#   Failed test 'Double Quoted Scalar'
#   at t/dump-heuristics.t line 10.
#          got: '--- 'A
# 
#   B
# 
#   C
# 
# '
# '
#     expected: '--- "A\nB\nC\n"
# '
# Looks like you failed 2 tests of 2.
t/dump-heuristics.t ...... 

...

#   Failed test 'Dumping Chinese hash works'
#   at t/utf8.t line 31.
#          got: '---
# Email: boss@opcafe.net
# 地址: 新竹市 300 石坊街 37-8 號
# 店名: OpCafé
# 時間: 11:01~23:59
# 電話: '03-5277806
# 
#   0991-100087
# 
# '
# '
#     expected: '---
# Email: boss@opcafe.net
# 地址: 新竹市 300 石坊街 37-8 號
# 店名: OpCafé
# 時間: 11:01~23:59
# 電話: "03-5277806\n0991-100087\n"
# '
# Looks like you failed 1 test of 8.
t/utf8.t ................. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/8 subtests 

The problem involves incorrectly handling newlines, but all this works and YAML::XS installs perfectly on my personal Perl, which I demonstrate below.
$ which perl ; perl -v | head -n 5 ; perl -MYAML::XS -e "print 'Yet Another Perl Hacker'"

/home/jacoby/perl5/perlbrew/perls/perl-5.24.0/bin/perl

This is perl 5, version 24, subversion 0 (v5.24.0) built for i686-linux
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2016, Larry Wall
Yet Another Perl Hacker

So, if it's Perl on Ubuntu on Windows, and the Perl works perfectly other places (including Strawberry Perl on the same Windows 10 machine (but only in PowerShell) ) and the Ubuntu works perfectly on other places, it must be the Windows, right?

That's what I thought, too. There's a GitHub repo for Ubuntu on Windows. Well, really, it's an issue tracker for Ubuntu on Windows, because there's nothing else in there. You can't clone and build your own. But, it does put the issue tracker in a place where the natural audience for Ubuntu on Windows would go. I put in an issue, but I think I might have asked wrong. I have cleaned up my issue some, but I see no sign that anyone else has seen it.

So, what is my next step? It isn't just YAML::XS; that's just a module I wanted at the time. It isn't just cpanm; I tried with cpan, too. It is just with Windows 10. Thoughts and suggestions from the greater Perl community?

2015/08/21

Perl Module for interacting with Globus

Ever heard of Globus?

No?

I'm kinda not surprised. It's a bit of an obscure thing.

Imagine you have research data. A collaborator wants to analyse it. Thing is, you're where you are and the collaborator is on the other side of the continent. There are many ways to move things, but many of them involve you having to be fairly technically sophisticated in order to make them work. Or your collaborator, who is an expert in your domain but not necessarily in file transfer. Plus, there's permissions and security and all that.

Or, you can use Globus.

We're changing how we handle data sharing here in the lab. We just moved file systems, moving to one that supports access control lists (ACLs), which means that, instead of playing games with symbolic links in order to share a researcher's work, we could simply setfacl -R -m 'u:user:r-x' and go from there.

That was fun, but because an ACL has to be set for every file, changes take a long time, so testing takes a long time, and updates take a long time. But it means that local researchers have access on the Big Iron.

But not everyone we want to share with is local. Some are on the other side of the continent. Thus Globus.

And, soon, Net::Globus.

It uses their first API, which is exposed via SSH. This is not my preferred interface, but I was able to get it to do the subset I needed fast enough, and I'll go back and make a thing work with their REST API in my copious free time. I mean, I could just use the Python API, right?

If you were to take my Globus.pm and copy it into your ~/lib directory, and you had the right stuff set up with SSH keys, it'd work. For the subset that isn't stubtastic, it's there.

I started of this project with dzil new, which is as much of Dist::Zilla as I know how to use. (I didn't start this paragraph intending an Elvis Costello reference, but there you go.) I know how to use git, how to make pull requests and merge them. My contribution graph is more white than green, but GitHub is not integral to my workflow, containing more my toys than my work. So, I'm pretty close to OK for the managing of a repo, but the steps toward getting it onto CPAN will involve me reading and asking a lot of stupid questions irc.perl.org.

Plus, the tests. Oh, man, the tests....

Perl is good because Perl is tested, and the tests I know how to write are tests I consider stupid. So, I expect to spend some time in chromatic's book of testing and in the tests for Net::Twitter and the like, trying to figure out what needs to be tested and how to write them.

So, I'm not ready to put it up to CPAN yet, but I see the path, because I finally have a module I could imagine someone else needing.

So, if this is a thing you think you have a use for, try it. If you think I did something stupid, I'm sure I did. Suggestions are great; pull requests are better. And thanks.