Day 16 – Yak Shaving for Fun and Profit (or How I Learned to Stop Worry and Love Perl 6)

History

A little over a year and a half ago I decided to write a radio station management application in Perl 5, I won’t bore you with the detailed reasons but it involved hubris and not liking having to modify an existing application that was written in multiple languages that I didn’t enjoy working with.   Most of the required parts were available on CPAN and the requirements were clear, so it was going along, albeit slowly.

Anyway in the early part of this year a couple of things coincided to make me consider that there would be more value in making the application in Perl 6:  the probability of a release before Christmas 2015 tied in with my original estimate that it would take approximately a year to finish the application, the features of Perl 6 that I  was aware of would lead to a nice neat design, and frankly it seemed like a cool idea to get in there with a large, possibly useful, application right as Perl 6 was beginning to enter the mainstream. I largely stopped working on the Perl 5 version and started looking at what I needed to be able to make it in Perl 6.  Of course this would prove to be even more hubristic and deluded than the original idea, but I didn’t know that at the time.

Entering the Yak Farm

It was immediately clear that I was going to have to write a lot more code for myself: at the turn of the year there were approximately 275 distributions in the Perl 6 modules ecosystem, whereas there are somewhere in the region of 30,000 modules on CPAN going back some twenty years. There were bits and pieces that I was going to need: Database access was coming along, there was a RabbitMQ client, people were working on web toolkits so some of the heavy lifting was already being worked on.

Having determined that I was going to have to write some modules it seemed that porting some of my existing CPAN modules might be a good way of getting up to speed and doing something useful into the bargain (though I’m not anticipating using any of them in the larger project.)

I thought I’d start with Linux::Cpuinfo because it was a fairly simple proposition on the face of it and I haven’t been happy with the AUTOLOAD that the Perl 5 version uses since a few months after I wrote it nearly fifteen years ago.

Straight into MOP Land

As it turns out losing the AUTOLOAD in Perl 6 was a fairly simple proposition, infact it could be replaced directly with a method called  “FALLBACK” in the class which gets called with the name of the required method as the first argument thus not requiring the not quite so nice global variable in Perl 5.  But it seemed nicer to take advantage of Perl 6’s Meta Object Protocol (MOP) and build a class based on the fields found in the /proc/cpuinfo on the fly, so I ended up with something like:

multi method cpu_class() {
    if not $!cpu_class.isa(Linux::Cpuinfo::Cpu) {
         my $class_name = 'Linux::Cpuinfo::Cpu::' ~ $!arch.tc;
         $!cpu_class := Metamodel::ClassHOW.new_type(name => $class_name);
         $!cpu_class.^add_parent(Linux::Cpuinfo::Cpu);
         $!cpu_class.^compose;
    }
    $!cpu_class;
}

And then add the fields from the data with:

submethod BUILD(:%!fields ) {
     for %!fields.keys -> $field {
         if not self.can($field) {
             self.^add_method($field, { %!fields{$field} } );
         }
     }
}

In the parent class of the newly made class, works really nicely. This is all going swimmingly, you can construct and manipulate classes using a documented interface without any external dependencies.

Losing the XS

The fact that the Perl 6 NativeCall interface allows you to bind functions defined in a dynamic library directly without requiring any external code didn’t seem to really help with the next couple of modules I chose to look at (Sys::Utmp and Sys::Lastlog) as the majority of the code in the Perl 5 XS files is actually dealing with the differences in various operating systems ideas of the respective interfaces as much as providing the XSUBs to be used by the Perl code.  As it happens this isn’t really so much of a problem as, with all the XSisms stripped out, the code can be compiled and linked to a dynamic library that can be used via NativeCall, even better someone had already made  LibraryMake that makes integrating all this into the standard build mechanisms really quite easy.  Infact it all proved so easy I made both of those modules in a few days rather than just the one that I had intended to do in the first place.

Anyhow by this point it was probably time to start on something that I might need in order to make the radio software, so I settled on Audio::Sndfile – it seemed generally useful and libsndfile is robust and well understood. And this is where the yak shaving really began to kick in. The module itself was really quite easy thanks to NativeCall despite the size of the interface and a subsequently fixed bug in the native arrays, but it occurred to me that automated testing of the module would be somewhat compromised if the dynamic library it depends on is not installed.

Testing Times

In order to facilitate the nicer testing of modules that depend on optionally installed dynamic libraries, I made LibraryCheck, though to be honest it was so easy I was surprised that no-one had made it before.  It exports a single subroutine that returns a boolean to indicate whether the specified library can be loaded or not, so in a test you might do something like:

use LibraryCheck;
if !library-exists('libsndfile') {
    skip "Won't test because no libsndfile", 10;
    exit;
}

As an aside you’ll notice that the extension to the dynamic library isn’t specified because NativeCall works that out for you (be it “.so”, “.dll”, “.dylib” or whatever.)

I’ve actually taken to putting the check in a Build.pm (which is used by panda if present to take user specified actions as part of the build):

class Build is Panda::Builder {
    method build($workdir) {
        if !library-exists('libsndfile') {
            say "Won't build because no libsndfile";
            die "You need to have libsndfile installed";
        }
        True;
     }
}

This has the effect of aborting the build before the tests are attempted, which for automated testing has the benefit of not showing false positives where the tests could not be attempted.

Of course a radio software doesn’t only need to read audio files, it really also should be able to stream audio out to listeners, so I next decided to make Audio::Libshout which binds the streaming source client library for Icecast libshout.  Not only does the testing of this depend on the dynamic library, but also requires the network service supplied by Icecast, so it would be nice to check whether the service was actually available before performing some of the tests.  So to this end I made CheckSocket which does exactly that using what is actually a fairly common pattern in tests for network services. It can be used in a similar fashion to LibraryCheck:

use CheckSocket;
if not check-socket($port, $host) {
    diag "not performing live tests as no icecast server";
    skip-rest "no icecast server";
    exit;
}

I’ve subsequently added it to at least one other author’s tests, it nicely encapsulates a pattern which would otherwise be ten or so lines of boilerplate that would have to be copied and pasted into each test file.

Enter the Gates of Trait

A feature of the implementation of libshout is that it has an initialisation function that returns an opaque “handle” pointer, that gets passed to the other functions of the library, in a sort of object oriented fashion, additionally it provides getter and setter functions for all of the parameters, these would best be modelled as read/write accessors in a Perl 6 class, but there would be a lot of boilerplate code to write these out by hand.  Having looked at a number of similar libraries I concluded that this may be a common pattern, so I wrote AccessorFacade to encapsulate the pattern.

AccessorFacade is implemented as a Perl 6 trait (I won’t explain “trait” here as a previous advent post has already done that,)  it allowed me to turn:

     sub shout_set_host(Shout, Str) returns int32 is native('libshout') { * }
     sub shout_get_host(Shout) returns Str is native('libshout') { * }

     method host() is rw {
         Proxy.new(
                    FETCH => sub ($) {
                                       shout_get_host(self);
                    },
                    STORE => sub ($, $host is copy ) {
                        explicitly-manage($host);
                        shout_set_host(self, $host);
                    }
         );
     }

Of which there may  be over a dozen or so, into:

sub shout_set_host(Shout, Str) returns int32 is native('libshout') { * } 
sub shout_get_host(Shout) returns Str is native('libshout') { * }

method host() is rw is accessor-facade(&shout_set_host, &shout_get_host) { }

Thus saving tens of lines of boilerplate, copy and paste mistakes and simplified testing. It probably took me longer to write AccessorFacade nicely after I had worked out how to do traits, than I ended up doing for Audio::Libshout. Which is a result, as next up I decided I needed to write Audio::Encode::LameMP3 in order to stream audio data that wasn’t already MP3 encoded, and it transpired that the mp3lame library also used the getter/setter pattern that AccessorFacade targetted, having that library enabled me to finish it even quicker than anticipated.

Up to my Ears in Yaks

Digital audio data is typically represented by large arrays of  numbers and with the native bindings there is a lot of creating native CArrays of the correct size, copying Perl arrays to native arrays and copying native arrays to Perl arrays and so forth, this was clearly a generally useful pattern so I created NativeHelpers::Array to collect all the common use cases, refactored all the audio modules to use the subroutines it exports and found myself in the position of having written more modules to help me make the modules that I wanted to write than the modules I actually wanted to write.  So in order to get things in balance I wrote Audio::Convert::Samplerate that used some of the helpers above.

Around this time I decided that I probably needed to concentrate on some application infrastructure requirements rather than domain specific things if I wanted to get anywhere with the original plan and started on a logging framework that I had been thinking about for a while.  I immediately concluded that I needed a singleton logger object so I wrote Staticish.

Staticish is implemented as a class trait that basically does two things: it adds a role to the class that gives it a singleton constructor that will always only return the same object (which is created the first time it is called,) and applies a role to the classes MetaClass (the .HOW,) which will cause the methods (and public accessors) of the class to be wrapped such that if the method is called on the type object of the class the singleton object will be obtained from the constructor and it will be called on that instead, so you can do something like:

use Staticish;

class Foo is Static {
    has Str $.bar is rw;
}

Foo.bar = "There you go";
say Foo.bar; # > "There you go";

It does exactly what I need it to, but I still haven’t finished that logging framework.

All at Sea with JSON

Earlier in the year I had started working on a Couch DB interface module with a view to using a document rather than a relational database in the application which is part of the reason I had been helping to make the HTTP client modules do some of the things that were needed, but another part, for me at least, was the ability to round-trip a Perl 6 class marshalled to JSON and back again, or vice versa. Half of that already existed with JSON::Unmarshal so I proceeded to make JSON::Marshal to do the opposite and then JSON::Class which provides a role such that a class has  to-json and from-json methods, all so good so far and you can do:

 use JSON::Class;

 class Something does JSON::Class {
     has Str $.foo;
 }

 my Something $something = Something.from-json('{ "foo" : "stuff" }');
 my Str $json = $something.to-json(); # -> '{ "foo" : "stuff" }'

But it needed some real world application to test it in, and fortunately this presented itself in the Perl 6 ecosystem itself: the META6.json that is crucial to the proper installation of a module. It is quite easy to mis-type if you are manually creating JSON, so META6 which can read and write the META6 files using JSON::Class and Test::META which can provide a sanity test of the meta file seemed like a good idea. Almost inevitably JSON::Unmarshal and JSON::Marshal needed to be revisited to allow custom marshallers/un-marshallers to be defined, (using traits, natch,) for certain attributes.

I had actually already started making JSON::Infer a year ago in Perl 5 for reasons that I could make an entire post about, but having already started down the JSON path it seemed easy to finish in Perl 6 and use JSON::Class in the created classes to allow the creation of JSON web service API classes easily: unfortunately JSON::Class (or rather its depencies) didn’t even survive the encounter with the test data, so I made JSON::Name so JSON object attribute who’s name wasn’t a valid Perl 6 identifier could be marshalled or unmarshalled correctly.  All fixed up this gave me the impetus to finish the WebService::Soundcloud which I had been playing with for a year or so.  I still haven’t finished the CouchDB interface.

Nowhere near to Land

So now toward the end of the year, there are 478 modules in the ecosystem and somehow I have made 29 of them and I haven’t even started to make the application that I set out to make at the beginning of the year, but I’m happy with that as hopefully I’ve made some modules that may be useful to someone else, I’ve become a confident Perl 6 programmer and the things I’ve learned have helped improve the documentation and possibly some of the code. I will finish the radio software in Perl 6 and I still know I have a lot of software to write but it becomes easier every day.  So if you’re thinking of making an application in Perl 6 yourself enjoy the journey more than you may be frustrated by not reaching the destination when you expected as you are probably travelling a path along which few have been before.