Day 20 – Bridging the gap

Perl 6 arrived last year, after quite a time, I might say! It promised a number of things, which we awaited eagerly. Some it delivered (a solid and sane threading model, for example), some it delivered in perhaps unexpected ways (it was ready by Christmas, after all), but some of them didn’t end up quite how we expected them to. One of those things is Perl 5 interoperability.

It should come as no surprise that CPAN, our legendary repository of modules for Perl 5, with readymade, well documented and thoroughly tested solutions for just about anything you can imagine[1] is quite unlikely to get ported to a completely new language. The bold move to drop backwards compatibility between Perl 5 and Perl 6 was necessary, but it came at a price: the pride of our community, the thing we used to call our language with Perl 5 itself being merely a VM to run it, our bread and butter is now doomed as a mere historical artifact, a legacy we may never live up to, but in the brave, new world of Perl 6 hardly useful anymore. What a loss that would be! Good news is that one of the assumptions of Perl 6 was that it’ll be capable of running Perl 5 code alongside Perl 6 code, loading Perl 5 modules and using them from Perl 6, among other things. Entire power of “good, old” CPAN available at your fingertips, dear reader. Bad news is, however, that this particular functionality we have not entirely get.

The original spec…ulation[2] documents said that you should be able to start a Perl 5 block of code in your Perl 6 code and expect it to work, as so:

grammar Shiny-New-Perl6-Things {
    ...
}

{
    use v5;

    sub good-old-perl5-things($$&) { ... }
}

That requires the Perl 6 compiler to be able to parse Perl 5 (yes, yes, I know; thankfully, the speculation also allows for a “well-behaved subset (…) much like PPI” :)) and execute it, providing interoperability between one and the other. Admirable as it is, it turns out to be much more complicated than it looks to be (and if it does look simple to you, remember to hug a Perl 6 core hacker next time you meet one). It doesn’t mean that there is no effort to make it reality (v5 being one of them, check it out!), but it’s not quite the promised land we were… promised. Yet, I hope! But now is now, and work needs to be done. So what are we left with?

Well, one of the things Perl 6 did deliver marvelously is the foreign function interface, most often used as a way to call C code from Perl 6. So some of us sat down and thought: well, Perl 5 is embeddable and is available to be ran as a C library, so to say. Perl 6 can call functions from C, C can execute Perl 5 code, what’s really stopping us from putting the two and two together? Quite a bit of hard work, but the goal is worthy, and smart, hardworking people is something we have quite in handy in our community. And so, Inline::Perl5 was born.

~~~

The simplest thing are hardly the most exciting ones, but let’s start somewhere:

use Inline::Perl5; # I will be skipping this from now on

my $p5 = Inline::Perl5.new;
$p5.run('print "Hello, older sister!\n"');

Or in a more Perl6-y way (and shorter, so better, right?):

EVAL 'print "Hello, older sister!\n"', :lang<Perl5>;

“But tadzik, how is that better than shelling out and having the external Perl 5 process do some predetermined thing?” Ah, it is better though: we can already drag the results out of the Perl 5 land and get it as a proper, Perl 6 object:

use Inline::Perl5;

my %thing = EVAL 'my %stuff = split(/[=,]/, "foo=bar,baz=beta"); \%stuff', :lang<Perl5>;
for %thing.kv -> $k, $v {
    say "$k => $v"
}

When a little elf saw this, it shouted “woo! Does that mean I can stuff whatever complex magic I want in that EVAL block and I’ll get back a working Perl 6 object?” Why yes, little elf. You can load modules, create objects and pass them around between one Perl and the other, and they’ll work just as you think they will. Check this hot stuff out:

my $mech = EVAL ' use WWW::Mechanize; WWW::Mechanize->new()', :lang<Perl5>;
say $mech.WHAT; # Perl5Object
$mech.get("https://perl6.org");

“Holy Moly!” the little elf could hardly contain itself. This means that the dream of having CPAN still available is still there, still reachable, and as usable as always! Can things get any better?

Hah! Why, I’m glad you asked. Remember how we made EVAL, the Perl 6 mechanism work in the Perl 5-augmented way and produce meaningful results? Syntactic magic is not a Perl 5 exclusive thing, you know. What makes you think we can’t teach use and alike to cooperate with Inline::Perl5 as well?

“Wait, you don’t mean…”

Oh yes, little elf. Yes I do:

use WWW::Mechanize:from<Perl5>;

my $mech = WWW::Mechanize.new;
$mech.get("https://perl6.org");

You don’t always see an elf cry, but when you do, it’s the tears of joy. Just like the ones I saw now, reflected in the comforting, blue-ish[3] light of the computer screen. It’s all there, integrated almost seamlessly, just as the speculations promised. The only price you pay is one use statement and some annotations on the ones that come from The World That Used To Be, and the rest works perfectly fine. All the power of CPAN in all the delicious wrapping paper of the new, shiny Christmas gift of a language. Little elf is a practical little being though, so it immediately started looking for edge cases that’ll prevent its real-world code from being utilized in this way. “I can create objects from Perl 6, call methods from Perl 6, get results back… as long as it can be represented with an object and method calls there’s is no limit to what I can do; in the worst case I can always wrap the good old Perl 5 module in something that plays nice with our limitations…”

I raise my eyebrow, Spock-like. “When would you need to do that?”

“Oh, you know”, started the little elf, “when an exception gets thrown for example.”

We almost couldn’t believe our ears when we heard Santa himself mutter “LOL” from behind his screen. We looked in confusion as he sent us this snippet:

my $mech = WWW::Mechanize.new;
try {
    $mech.get("xmas://perl6.org");
    CATCH {
        default {
            say $_.perl;
        }
    }
}
# Output: X::AdHoc.new(payload => "Error GETing xmas://perl6.org: Protocol scheme 'xmas' is not supported at -e line 0.\n")

The little elf looked almost indignant. “Oh for dancing reindeers, is there anything this thing can’t do!?”

“Like what?” Rudolph asked casually, passing by on its four legs.

“Gee, I don’t think I… ah!” it exclaimed suddenly, finding in its memory a particularly complicated piece of code running the facility at this very moment. “I guess if I need to create objects that are subclasses of existing Perl 5 classes I’m a bit out of luck, am I not?”

“What makes you think that wouldn’t just work as you expect it to?”

“I… heh, I guess I should’ve tried it first. So all the complex things, DBIx::Class, Catalyst, they can all work with this just fine?”

“Been there, done that” muttered Santa from behind his screen again, while the servers hummed peacefully running the now part Perl 5, part Perl 6 production code in the gift factory.

“Well, the future sure does seem bright, does it not?”

“Yes it does”, I replied. “It really does.”

Light years away, a star shone bright, and while it’ll take us a few more days to actually see it above our heads, it is already there, sooner than we thought it would be.

~~~

We may not have gotten the seamless interop we asked for, but for all practical purposes our marvelous Perl 5 legacy is far from gone. It may be almost surprising how much you can already get done with it. Those few battle-tested modules that need glueing together? This nasty script that needs refactoring so badly it may as well end up being a full-blown rewrite? That little project that you dreamed of using Perl 6 for, but never expected to have all the dependencies available (and good, and fast, and proved to work)? It’s time to try it. Your star may already be shining.

[1] I swear I remember seeing a module that existed for the sole purpose of uploading the My Little Pony fanfiction to an appropriate place. If you can help me find it, there’s a prize, why the heck not!

[2] We try very hard to not call it specification :)

[3] Yes, we know about Redshift in the Santa Claus Magical factory, but it’s December, crunch time, we can’t afford to go to bed quite at the time we wish we would be able to. Compromises had to be made.

Day 8 – Grammars generating grammars

By now you have probably gotten used to the prefix “meta” appearing here and there in the Perl 6 world. Metaclasses, metaobjects, metaoperators, the mythical Meta-Object Protocol. Sounds like nothing scary, all good and familiar and you’ve seen it all, eh? Nothing further from the truth! Today, on the Perl 6 Advent Calendar we’re going full meta. We’re going to have grammars that parse grammars then generate grammars that are going to use to parse grammars. I’ll let this sink in for a moment while you scroll down for the next paragraph.

Grammars are hands-down one of the killer features of Perl 6. Taking Perl’s no less than legendary ability to process text and taking it to the next level. Regular expressions, to many people’s disappointment are just what they say they are: regular (well, regular regular expressions do. Perl regexes are a bit of a different beast, but that’s a material for another story). They parse regular, as opposed to, say, context-free languages, to the neverending disappointment of all the people who would just love to parse XML with them, like in that everyone’s favourite Stackoverflow answer. Say no more to silly theoretical limitations of language theory though, since now we have grammars which are all we ever missed in regexes: readability, composability and, of course, ability to parse even Perl 6 itself — and if that doesn’t sell its absolute power, I don’t know what does.

Writing parsers for predefined grammars (say, in Bachus-Naur Form) was always a bit of a dull job, almost as exciting as copypasting stuff. If you ever sat down and wrote a parser from scratch (perhaps while going through the excellent Let’s Build a Compiler book), you probably recognize the pattern all too well: take a single rule from your grammar, write a subroutine for it, have it call (perhaps recursively) other subroutines similarly defined for other grammar rules, rinse, repeat. Well say no more to that now that we have Perl 6 grammars! In this wonderful new world we need not write subroutines for every token to get the work done. Now we write a grammar class, where we put tokens, rules and regexes for every symbol in our grammar, and inside them we write regexes (or code) that refer to (perhaps recursively) other symbols in our Perl 6 grammar. Now, if you ever went through both of those alternatives, you will definitely realize how massive of a convenience the grammars are in Perl 6. Instead of painstakingly cranking out repetitive and error-prone code we have a wonderful, declarative way to specify our language, with an impressive collection of utility to get most of our common, boring work out of the way.

But what if we already have a grammar, specified perhaps in the previously mentioned BNF? What we do then is carefully retype the existing grammar (parsing it in our head, actually) into our new, shiny Perl 6 grammar that represents the exact same thing, but has the clear advantage of actually being executable code. A fair deal, you could say. For most people, no big deal at all. We are not most people. We are programmers. We have the resources. The will. To make these grammars count! Our job revolves around building things that will do the work for us so we don’t have to. To take the well-defined specifications and turn them into tools that work for us, or for themselves. Tools that take the repetitive and automatable parts of our work the way. Why should building parsers be any different? Why, I’m glad you asked.

The wonderful thing about the Perl 6 grammars is that they are no more magical than any other element of the language. Just as classes are first class citizen that we can introspect, augment and build programatically, so are grammars. In fact, you can look at the source code of the compiler itself and notice that grammars are nothing else than a specialized kind of classes. They follow the same rules as classes do, allowing us to create them on the fly, add tokens to them on the fly and eventually finalize them in order to have a proper, instantiatable class object. So now that we can parse BNF grammars (since they’re just ordinary text) and create Perl 6 grammars from code, let’s put those pieces together and write us something that will save us the effort of converting a BNF grammar to Perl 6 grammar manually.

The grammar for a BNF grammar

grammar Grammar::BNF {
    token TOP { \s* <rule>+ \s* }

    token rule {
        <opt-ws> '<' <rule-name> '>' <opt-ws> '::=' <opt-ws> <expression> <line-end>
    }

    token expression {
        <list-of-terms> +% [\s* '|' <opt-ws>]
    }

    token term {
        <literal> | '<' <rule-name> '>'
    }

    token list-of-terms { <term> +% <opt-ws> }

    token rule-name { <-[>]>+ }

    token opt-ws { \h* }

    token line-end { [ <opt-ws> \n ]+ }

    token literal { '"' <-["]>* '"' | "'" <-[']>* "'" }

    ...
}

The interesting stuff happens in the three, almost-topmost tokens. rule is the core building block of a BNF grammar: a <symbol> ::= <expression> pair, followed by a new line. The entire grammar is no more than a list of those. Each expression is a list of terms, or possibly and alternative of them. Each term is either a literal, or a symbol name surrounded by angle brackets. Easy enough! That covers the parsing part. Let’s look at the generating itself. We do have a built-in mechanism of “doing stuff for each token in the grammar”, in the form of Actions, so let’s go ahead and use that:

my class Actions {
    has $.name = 'BNFGrammar';
    method TOP($/) {
        my $grmr := Metamodel::GrammarHOW.new_type(:$.name);
        $grmr.^add_method('TOP',
            EVAL 'token { <' ~ $<rule>[0].ast.key ~ '> }');
        for $<rule>.map(*.ast) -> $rule {
            $grmr.^add_method($rule.key, $rule.value);
        }
        $grmr.^compose;
        make $grmr;
    }

    method expression($/) {
        make EVAL 'token { ' ~ ~$/ ~ ' }';
    }

    method rule($/) {
        make ~$<rule-name> => $<expression>.ast;
    }
}

The TOP method is definitely the most magical and scary, so let’s tackle that first to make the rest of the stuff look trivial in comparison. Basically, three things happen there:

1. We create a new grammar, as a new Perl 6 type
2. We add tokens to it using the `^add_method` method
3. We finalize the grammar using the `^compose` method

While Perl 6 specifies that the token named TOP is where the parsing starts, in BNF the first rule is always the starting point. To adapt one to the other, we craft a phony TOP token which just calls the first rule specfied in the BNF grammar. Unavoidably, the scary and discouraging EVAL catches our attention, as if it said “horrible stuff happens here!” It’s not entirely wrong when it says that, but since we have no other way of programmatically constructing the individual regexes (that I know of), we’ll have to accept this little discomfort in the name of the Greater Good, and look a little bit closer at what we’re actually EVALing in a moment.

After TOP we proceed to add the rest of the BNF rules to our grammar, this time preserving their original names, then ^compose() the whole thing and finally make it the result of the parsing: a readymade parser class.

In the expression method we glue the parsed BNF elements together in order to produce valid Perl 6 code. This turns out to be pretty easy, since both separate symbols with whitespace, alternate them with the pipe character and sorround symbol names with angle brackets. So for a rule that looks like this:

<foo> ::= 'bar' | <baz>

The Perl 6 code that we EVAL turns becomes:

token { 'bar' | <baz> }

Since we already validated in the grammar part of our code that the BNF we parse is correct, there’s nothing stopping us from literally pasting the entire expression into our code and wrap it in a token { }, so let’s go ahead and do just that.

Last but not least, for each BNF rule we parse we produce a nice Pair, so our TOP method has an easier times processing each of them.

Seems like we’re about done here, but just for the users’ convenience, let’s write a nice method that takes a BNF grammar and produces a ready to use type object for us. As we remember, grammars are just classes, so there’s nothing stopping us from just adding it straight to our grammar:

grammar Grammar::BNF {
    ...

    method generate($source, :$name = 'BNFGrammar') {
        my $actions = Actions.new(:$name);
        my $ret = self.new.parse($source, :$actions).ast;
        return $ret.WHAT;
    }
}

Looks good from here! Before you start copypasting all this into your own projects, remember that Grammar::BNF is a Perl 6 module available in your Perl 6 Module Ecosystem, installable with your favourite module manager. It also ships with some goodies like a Perl 6 slang, allowing you to write BNF grammars straight in your Perl 6 code (as opposed to including them as strings), or parsing the more powerful (and perhaps more widespread) ABNF grammars as well.

If you indeed took the time for the beginning of this post to sink in, you may remember that I promised that we’re going to have grammars (a-one) that parse grammars (a-two) then generate grammars (a-three) that are going to use to parse grammars (a-four). So far we’ve seen the BNF::Grammar grammar (that’s our a-one), that parses a BNF grammar (that’s our a-two), generates a Perl 6 grammar in a form of a type object (that’s a-three) and… that’s it. We’re still missing the last part, using this whole thing to parse grammars. We’ve only gone 75%-meta, and that is just not good enough in this day and age. Why stop now? Why not take a BNF grammar of a BNF grammar, parse that with a Perl 6 grammar and use the resulting Perl 6 BNF grammar to parse our original BNF grammar of a BNF grammar? Wouldn’t that be sweet? Sure it will! That is, however, left as an exercise for you, my dear readers. After all, how fair would it be if I had all the fun while you just sit there and watch? Would you like it if you opened the little paper window in your advent calendar only to find a note saying. „There was a chocolate here for you. I ate it. It was truly delivious?” Me neither! There’s a BNF grammar for BNF both on Wikipedia and in Grammar::BNF’s very own test suite; the latter even includes a little breadcrumb that can help you with your adventure. I eagerly await thy results, and as always, thank you all for reading and I wish you a wonderful advent!

Day 2 – Rakudobrew

Traditionally in Perl, and probably in any other scripting language (I’m going to use the term “scripting” to describe “one that requires a runtime”, bear with me) there’s quite a big discrepancy between the latest released version of the toolchain and the one that your OS distribution ships (if it ships one at all!). My Ubuntu for instance, the freshly updated 14.10 not only offers me to install a nine-month-old Rakudo 2014.03, but also one running on top of Parrot, which I don’t remember voluntarily running anytime during 2014, frankly. Even if you look at languages with a longer release cycle, like Perl 5 which releases every year, system distributions shipping old versions is still a big deal: modern Perl modules still often maintain backwards compatibility with Perl 5.8 released in 2002, because that’s what some versions of CentOS comes with.

A popular idea is to say that system Perl (or system Python, system Ruby or whatever floats your boat) is for the system itself, while we programmers can freely uses whatever version we wish as long as we install it and maintain it in our home directory and not mess with system stuff. Perl 5 people usually recommend either Perlbrew or plenv for that; in Perl 6 rakudobrew is often said to be the easiest way to stay up-to-date with recent developments.

Rakudobrew is a commandline tool that makes it easy to install and update different flavours of Rakudo, as well as switch between them any time you want to. It also updates Panda, the module installer, after every update, along with all modules you had installed before (precompiled modules need to be updated after every update of Rakudo).

I made an effort to keep rakudobrew reasonably free of dependencies: it only requires Perl 5 and git, which are roughly what’s required to build Rakudo anyway. It’s known to work well on Linuxes and OSX, but I’ve also recently got some pull requests to improve Windows support (which apparently worked for a while by itself, without me knowing about it. Cool!) so it may happen to be useful to you, our Windows-running readers just as well.

Rakudobrew requires almost zero setup to get running; the only thing you need to do is to get it from Github (either by cloning it or downloading a tarball from the website), and get your system to recognize its bin/ subdirectory as part of your PATH. What I’m usually going for on my machines is the following:

$ git clone https://github.com/tadzik/rakudobrew ~/.rakudobrew
$ export PATH=~/.rakudobrew/bin:$PATH
$ echo 'export PATH=~/.rakudobrew/bin:$PATH' >> ~/.zshrc
$ # or ~/.bashrc or whatever

After that it requires nothing like ‘rakudobrew init’ or anything; it’s ready to get useful. The most basic thing you need from it is of course installing Rakudo:

$ rakudobrew build moar

The above command will download and build Rakudo on MoarVM, the latest version available from Github. For extra stability you may want to install the latest available release and just stick to that:

$ rakudobrew build moar 2014.11

This will take a while, but after a while it should finish saying “Done, moar-2014.11 built.” From that point on you are ready to go; if you adjusted your PATH correctly, the “perl6” command will be available for you to use, and you’re ready to hack!

That said, you may want something more from life than just the bare compiler. The “useful and usable” Perl 6 distribution called Rakudo Star doesn’t have its own Starbrew, but that doesn’t mean you can’t eat the cake and still have it. Let’s also install Panda, the module installer, and whatever Rakudo Star usually ships.

$ rakudobrew build panda
$ panda install Task::Star

This gets you not only the module installer (which is instantly ready to go, just as shown above), but also all the widely appreciated goodies like: NativeCall, Term::ANSIColor, DBIish or Grammar::Debugger. Life’s good!

But what if you want more from life than just MoarVM? Perhaps you want to try out the famous JVM interop? Couldn’t be easier!

$ rakudobrew build jvm
$ rakudobrew switch jvm

Contrary to what it may suggest the above commands don’t build the JVM itself (you’re expected to have that installed), but the version of Rakudo running on the Java Virtual Machine. Note that you’ll still have to install panda and all the modules you want; those are installed per every instance of Rakudo you built. In case you got lost in all the versions you installed, the list command is there to help you again:

$ rakudobrew list
  jvm-HEAD
  moar-2014.11
* moar-HEAD

The asterisk is marking the one currently active (‘rakudobrew current’ can also tell you that). You don’t need to type the full name either; both ‘rakudobrew switch jvm’ and ‘rakudobrew switch moar-H’ will work just fine.

In case you want to get rid of something you installed (perhaps you wanted to test your code on some older version of Rakudo, or see how slow things were back in the days (the further you reach the more amazing it gets)), it’s as easy as removing a subdirectory from rakudobrew’s directory:

$ rm -rf ~/.rakudobrew/jvm-HEAD

Or in case you want to get rid of the entire rakudobrew for any reason, you don’t have to be afraid of any globally installed leftovers: there aren’t any. It’s all self-contained, so as simple as

$ rm -rf ~/.rakudobrew

And poof! Gone. Before you get rid of it though, you may want to keep your still warm Rakudo around to try out more interesting stuff that our Advent Calendar will have to show you in the coming weeks. Stay tuned!

Day 19 – Gather and/or Coroutines

Today I’ll write about coroutines, gather-take and why they are as much fun as one another. But since it’s all about manipulating control flow, I took the liberty to reorganize the control flow of this advent post, so coroutines will finally appear somewhere at the end of it. In the meantime I’ll introduce the backstory, the problems that coroutines solved and how it looks from the Perl 6 kitchen.

LWP::Simple is all fun and games, but sometimes you can’t afford to wait for the result to come. It would make sense to say “fetch me this webpage and drop me a note when you’re done with it”. That’s non trivial though; LWP::Simple is a black box, which we tell “get() this, get() that” and it gives us the result back. There is no possible way to intercept the internal data it sends there and around. Or is there?

If you look at Perl 5’s AnyEvent::HTTP, you’ll see that it reimplemented the entire HTTP client to have it non-blocking. Let’s see if we can do better than that.

First thing, where does LWP::Simple actually block? Behind our backs it uses the built-in IO::Socket::INET class. When it wants data from it, it calls .read() or .recv() and patiently waits until they’re done. If only we could somehow make it not rely on those two directly, hmm…

„I know!”, a gemstone-fascinated person would say, „We can monkey-patch IO::Socket::INET”. And then we have two problems. No, we’ll go the other way, and follow the glorious path of Dependency Injection.

That sounds a bit scary. I’ve heard about as many definitions of Dependency Injection as many people I know. The general idea is to not create objects inside other objects directly; it should be possible to supply them from the outside. I like to compare it to elimination of „magic constants”. No one likes those; if you think of classes as another kind of magic constants which may appear in somebody else’s code, this is pretty much what this is about. In our case it looks like this:

# LWP::Simple make_request
my IO::Socket::INET $sock .= new(:$host, :$port);

There we go. “IO::Socket::INET” is the magic constant here; if you want to use a different thing, you’re doomed. Let’s mangle it for a bit and allow the socket class to come from the outside.

We’ll add an attribute to LWP::Simple, let’s call it $!socketclass

has $.socketclass = IO::Socket::INET;

If we don’t supply any, it will just fallback to IO::Socket::INET, which is a sensible default. Then, instead of the previous .new() call, we do

my $sock = $!socketclass.new(:$host, :$port);

The actual patch (https://github.com/tadzik/perl6-lwp-simple/commit/93c182ac2) is a bit more complicated, as LWP::Simple supports calling get() not only on constructed objects but also on type objects, which have no attributes set, but we only care about the part shown above. We have an attribute $!socketclass, which defaults to IO::Socket::INET but we’re free to supply another class – dependency-inject it. Cool! So in the end it’ll look like this:

class Fakesocket is IO::Socket::INET {
    method recv($) {
        note 'We intercepted recv()';
        callsame;
    }

    method read($) {
        note 'We intercepted read()';
        callsame;
    }
}

# later
my $lwp = LWP::Simple.new(socketclass => Fakesocket);

And so our $lwp is a fine-crafted LWP::Simple which could, theorically, give the control flow back to us while it waits for read() and recv() to finish. So, how about we put theory into practice?

Here start the actual coroutines, sorry for being late :)

What do we really need in our modified recv() and read()? We need a way to say „yeah, if you could just stop executing and give time to someone else, that would be great.” Oh no, but we have no threads! Luckily, we don’t need any. Remember lazy lists?

my @a := gather { for 1..* -> $n { take $n } }

So on one hand we run an infinite for loop, and on the other we have a way to say „give back what you’ve come up with, I’ll catch up with you later”. That’s what take() does: it temporarily jumps out of the gather block, and is ready to get back to it whenever you want it. Do I hear the sound of puzzles clicking together? That’s exactly what we need! Jump out of the execution flow and wait until we’re asked to continue.

class Fakesocket is IO::Socket::INET {
    method recv($) {
        take 1;
        callsame;
    }

    method read($) {
        take 1;
        callsame;
    }
}

# later
my @a := gather {
    $lwp.get("http://jigsaw.w3.org/HTTP/300/301.html");
    take "done";
}

# give time to LWP::Simple, piece by piece
while ~@a.shift ne "done" {
    say "The coroutine is still running"
}
say "Yay, done!";

There we go! We just turned LWP::Simple into a non-blocking beast, using almost no black magic at all! Ain’t that cool.

We now know enough to create some syntactic sugar around it all. Everyone likes sugar.

module Coroutines;
my @coroutines;
enum CoroStatus <still_going done>;

sub async(&coroutine) is export {
    @coroutines.push($(gather {
        &coroutine();
        take CoroStatus::done;
    }));
}

#= must be called from inside a coroutine
sub yield is export {
    take CoroStatus::still_going;
}

#= should be called from mainline code
sub schedule is export {
    return unless +@coroutines;
    my $r = @coroutines.shift;
    if $r.shift ~~ CoroStatus::still_going {
        @coroutines.push($r);
    }
}

We maintain a list of coroutines currently running. Our async() sub just puts a block of code in the execution queue. Then every call to yield() will make it jump back to the mainline code. schedule(), on the other hand, will pick the first available coroutine to be run and will give it some time to do whatever it wants.

Now, let us wait for the beginning of the post to catch up.

Native libraries, native objects

Last year flussence++ wrote a nice post about writing XMMS bindings for Perl 6 using the Native Call Interface. It has improved a bit since then, (at least NCI, I don’t know about XMMS), so let’s show it off a bit.

To run the examples below you need a NativeCall module installed. Then add use NativeCall; at the top of the file.

Previously, we were carefully writing all the C subs we needed to use and then usually writing some Perl 6 class which wrapped it in a nice, familiar interface. That doesn’t change much, except that now a class is not really an interface for some C-level data structure. Thanks to the new metamodel we can now make our class to actually be a C-level data structure, at least under the hood. Consider a class representing a connection to Music Player Daemon:

    class Connection is repr('CPointer') {
        sub mpd_connection_new(Str $host, Int $port)
            returns Connection
            is native('libmpdclient.so') {}
        sub mpd_connection_free()
            is native('libmpdclient.so') {}
        method new(Str $host, Int $port) {
            self.bless(mpd_connection_new($host, $port))
        }
        method DESTROY {
            mpd_connection_free(self)
        }
    }

The first line does not necesarilly look familiar. The is repr trait tells the compiler that the internal representation of the class Connection is a C pointer. It still is a fully functional Perl 6 type, which we can use in method signatures or wherever (as seen in the lines below).

We then declare some native fuctions we’re going to use. It’s quite convenient to put them inside the class body, so they don’t pollute the namespace and don’t confuse the user. What we are really exposing here is the new method, which uses bless to set the object’s internal representation to what mpd_connection_new has returned. From now on our object is a Perl 6 level object, while under the hood being a mere C pointer. In method DESTROY we just pass self to another native function, mpd_connection_free, without the need to unbox it or whatever. The NativeCall module will just extract its internal representation and pass it around. Ain’t that neat?

Let’s see some bigger example. We’ll use taglib library to extract the metadata about some music files lying around. Let’s see the Tag class first:

    class Tag is repr('CPointer') {
        sub taglib_tag_title(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_artist(Tag) returns Str is native('libtag_c.so') {}
        sub taglib_tag_album(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_genre(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_year(Tag)   returns Int is native('libtag_c.so') {}
        sub taglib_tag_track(Tag)  returns Int is native('libtag_c.so') {}
        sub taglib_tag_free_strings(Tag)       is native('libtag_c.so') {}

        method title  { taglib_tag_title(self)  }
        method artist { taglib_tag_artist(self) }
        method album  { taglib_tag_album(self)  }
        method genre  { taglib_tag_genre(self)  }
        method year   { taglib_tag_year(self)   }
        method track  { taglib_tag_track(self)  }

        method free   { taglib_tag_free_strings(self) }
    }

That one is pretty boring: plenty of native functions, and plenty of methods being exactly the same things. You may have noticed the lack of new: how are we going to get an object and read our precious tags? In taglib, the actual Tag object is obtained from a Tag_File object first. Why didn’t we implement it first? Well, it’s going to have a method returning the Tag object shown above, so it was convenient to declare it first.

    class TagFile is repr('CPointer') {
        sub taglib_file_new(Str) returns TagFile is native('libtag_c.so') {}
        sub taglib_file_free(TagFile)            is native('libtag_c.so') {}
        sub taglib_file_tag(TagFile) returns Tag is native('libtag_c.so') {}
        sub taglib_file_is_valid(TagFile) returns Int
            is native('libtag_c.so') {}

        method new(Str $filename) {
            unless $filename.IO.e {
                die "File '$filename' not found"
            }
            my $self = self.bless(taglib_file_new($filename));
            unless taglib_file_is_valid($self) {
                taglib_file_free(self);
                die "'$filename' is invalid"
            }
            return $self;
        }

        method tag  { taglib_file_tag(self)  }

        method free { taglib_file_free(self) }
    }

Note how we use native functions in new to check for exceptional situations and react in an appropriately Perl 6 way. Now we only have to write a simple MAIN before we can test it on our favourite music files.

    sub MAIN($filename) {
        my $file = TagFile.new($filename);
        my $tag  = $file.tag;
        say 'Artist: ', $tag.artist;
        say 'Title:  ', $tag.title;
        say 'Album:  ', $tag.album;
        say 'Year:   ', $tag.year;

        $tag.free;
        $file.free;
    }

Live demo! Everyone loves live demos.

    $ perl6 taginfo.pl some-track.mp3
    Artist: Diablo Swing Orchestra
    Title:  Balrog Boogie
    Album:  The Butcher's Ballroom
    Year:   2009

Works like a charm. I promise I’ll wrap it up in some nice Audio::Tag module and release it on Github shortly.

Of course there’s more to do with NativeCall than just passing raw pointers around. You could, for example, declare it as a repr('CStruct') and access the struct field directly, as you would in good, old C. This is only partly implemented as for now though, but that shouldn’t stop you from experimenting and seeing what you can do before Christmas. Happy hacking!

Bailador — A small Dancer clone

Today we’ll write a simple Dancer clone in Perl 6. Simple also means Very Minimal — it will only recognize basic GET requests. Let’s look at how the simplest Dancer app possible looks like:

    get '/' => sub {
        "Hello World!"
    };
    dance;

So we need something to add routes to our app, and something to run it. Let’s take care of adding routes first. We’ll create an array to store all those, and thus get() will just add them to it.

    my @routes;
    sub get(Pair $x) is export {
        @routes.push: $x;
    }

In case you’re not familiar with the Pair thing, in Perl 6 a fat comma (=>) creates an actual data structure containing a key and a value. In this case, the key is just a string ‘/’, and the value is a subroutine.

Having @routes being a simple array of keys and values we can now write a simple dispatcher:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                return $r.value.();
            }
        }
        return "404";
    }

dispatch() takes a hash representing our environment, which contains the REQUEST_URI string, basing on which we’ll try to find an appropriate candidate to dispatch to.

The above example is also cheating a bit: it just returns a ‘404’ string instead of creating a proper HTTP response. Making it respond properly is left as an exercise for the reader (not the last one in this short article, I assure you :)).

Since we got that far already, writing our own dance() is a piece of cake. We’re going to call it baile() though. Why do we write all this in Spanish? If you can guess on which classes I was bored and wrote this thing on a piece of paper, then on the next YAPC I’ll show you the fastest possible way to tie a shoe. No kidding! But let’s finish this thing first.

Luckily we don’t need to write our own web server now. We have HTTP::Server::Simple::PSGI in Perl 6, which will do most of the hard work for us. The only thing we have to do is to create a PSGI app. In case you’ve never heard of it, a PSGI app is a mere subroutine, taking the environment as an argument, and returning an array of three things: an HTTP response code, an array of HTTP headers and a response body. Once we have our PSGI app ready, we just feed HTTP::Server::Simple::PSGI with it, and we’re good to go.

    sub baile is export {
        my $app = sub($env) {
            my $res = dispatch($env);
            return ['200', [ 'Content-Type' => 'text/plain' ], $res];
        }

        given HTTP::Server::Simple.PSGI.new {
            .host = 'localhost';
            .app($app);
            .run;
        }
    }

Yep, we’re cheating again and returning 200 no matter what. Remember the part about “an exercise for the reader?” You can think about it while celebrating a working Dancer clone.

But wait, there’s more!

Let’s look at our dispatch() once again:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                return $r.value.();
            }
        }
        return "404";
    }

You probably noticed that we’ve used ~~ — a smartmatching operator. Thanks to that, we can match REQUEST_URI against strings, but not only. Junctions will work fine too:

    get any('/h', '/help', '/halp') => sub {
        "A helpful help message"
    }

And regexes:

    get /greet\/(.+)/ => sub ($x) {
        "Welcome $x"
    }

The last one will need a bit of tweaking in dispatch(). Yes, ~~ does the regex matching for us, but we have to take care of passing the match results to the callback function. Let’s modify the if body then:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                if $/ {
                    return $r.value.(|$/.list);
                } else {
                    return $r.value.();
                }
            }
        }
        return "404";
    }

The if $/ part checks whether the match resulted in setting the Match object in the $/ variable. If it did, we flatten the Match to a list, and pass it to the callback function. We need a | prefix, so it becomes expanded to a parameter list instead of being passed as a mere array. From now on, the above example with greet will Just Work. Yay!

The Bailador code presented here is available in the Github repository. If you feel challenged by the “exercises for the reader”, or just want to make it a bit more proper Dancer port, you’re welcome to hack on it a bit and contribute. I hope I showed you how simple it is to write a simple, yet useful thing, and going with those simple steps we can hopefully get to something close to a full-blown Dancer port. Happy hacking, and have an appropriate amount of fun!

Documenting Perl 6

A wise man once said that programs must be written for people to read, and only incidentally for machines to execute. But aside from being read, your code is also going to be used by people, who don’t really want to dive into it to understand what it does. That’s where the documentation comes in.

In Perl 5 we had POD, which stands for Plain Old Documentation. In Perl 6 we have Pod, which is not really an abbreviation of anything. As its specification says, “Perl 6’s Pod is much more uniform, somewhat more compact, and considerably more expressive”. It has changed slightly compared to Perl 5 Pod, but most of the stuff remains the same, or at least very similar.

There are three main types of Pod blocks in Perl 6. Delimited blocks are probably the most obvious and simple ones:

    =begin pod

    <whatever Pod content we want>

    =end pod

Paragraph blocks are a bit more magical. They begin with =for, and end on the nearest blank line (as the name, Paragraph, suggest):

    my $piece = 'of perl 6 code'

    =for comment
    Here we put whatever we want.
    The compiler will not notice anyway.

    our $perl6 = 'code continues';

Abbreviated blocks are similar to Paragraph blocks. The leading = is followed immediately by a Pod block identifier and the content. Sounds familiar?

    =head1 Shoulders, Knees and Toes

That’s right, =head is nothing magical in Perl 6 Pod. That means you can write it also as a paragraph block

    =for head1
    Longer header
    than we usually write.

Or a full-blown delimited block

    =begin head1

    This header is longer than it should be

    =end head1

Any block can be written as a delimited block, paragraph block, or abbreviated block. No magic. Not all blocks are created equal, of course. =head will be treated differently than plain =pod. By whom? By the Pod renderer, of course, but also by the Perl 6 compiler itself. In Perl 6, Pod is not a second-class citizen, ignored during the program compilation. Pod in Perl 6 is a part of the code; along with parsing and constructing AST of the code to be executed, the compiler also parses and builds AST of all Pod blocks. They are then kept in the special $=pod variable, and can be inspected by the runtime:

    =begin pod

    Some pod content

    =end pod

    say $=pod[0].contents[0].contents;

The say line may look a little complicated. Content, of a content, of a what? What really happens, is that ‘Some pod content’ is parsed as an ordinary paragraph, and kept in the Pod::Block::Para object. The delimited block started with =begin pod becomes a Pod::Block::Named, and it can contain any number of child blocks. It’s also a first block in our example code, so it ends up in $=pod[0].

You now probably think “geez, how ugly is that. Who’s going to use it in this form”. Don’t worry. Frankly, I don’t expect anyone to use the AST directly. That’s what Pod renderers are useful for. Take for example Pod::To::Text:

    =begin pod

    =head1 A Heading!

    A paragraph! With many lines!

        An implicit code block!
        my $a = 5;

    =item A list!
    =item Of various things!

    =end pod

    DOC INIT {
        use Pod::To::Text;
        pod2text($=pod);
    }

Ran as it is, the code doesn’t produce any output. Why is it so? The DOC INIT block looks a little special. It gets run with every other INIT block, but also only when the --doc flag is passed to the compiler. Let’s take a look:

    $ perl6 --doc foo.pl
    A Heading!
    
    A paragraph! With many lines!
    
        An implicit code block!
        my $a = 5;
    
     * A list!
    
     * Of various things!

Actually, when no DOC INIT block exists in the code, the compiler generates a default DOC INIT, identical to the one in the example above. So you could really omit it, leaving only the Pod in the file, and perl6 --doc will produce exactly the same result.

But wait, there’s more!

I wrote about 3 types of Pod blocks, but there’s another one I didn’t talk about before. They are Declarator blocks, and they single purpose is to document the actual Perl 6 objects. Take a look.

    #| it's a sheep! really!
    class Sheep {
        
        #| produces a funny sound
        method bark {
            say "Actually, I don't think sheeps bark"
        }
    }

Every declarator block gets attached to the object which comes after it. It’s available in the .WHY attribute, so we can use it like this:

    say Sheep.WHY.contents;                      # it's a sheep! really!
    say Sheep.^find_method('bark').WHY.contents; # produces a funny sound

In this case we also don’t need to care about doing a ^find_method and all this for every piece of documentation we want to read. The mighty Pod::To::Text takes care about it too. If we run the Sheep code with --doc flag, we get the following:

    class Sheep: it's a sheep! really!

    method bark: produces a funny sound

The specification says it’s possible to document all the class attributes and all the arguments that methods or subroutines take. Unfortunately no Perl 6 implementation (that I know of) implements those yet. (UPDATE: Rakudo Perl 6 now implements this!)

There are dozens of Pod features that are not covered by this post, for example the formatting codes (<, > and so), or tables. If you’re interested take a look at Synopsis 26 (http://perlcabal.org/syn/S26.html). It’s actually written in Pod 6, and rendered by Pod::To::HTML. Not all features it describes are implemented yet, but most of them are (see the test suite linked below), and some modules are actually documented with it (Term::ANSIColor for example).

Some useful links:

Synopsis 26

Pod::To::Text source code

Term::ANSIColor documentation

Pod test suite (shows what Pod in Rakudo is capable of)

Happy documenting!

Day 12 – Smart matching

Remember the https://perl6advent.wordpress.com/2010/12/04/the-sequence-operator/ sequence operator? As the last argument it takes a limit, which makes the sequence generation stop. For example:

    1, 2, 4 ... 32;         # 1 2 4 8 16 32
    1, 2, 4 ... * > 10;     # 1 2 4 8 16

You can see that in the first case, numerical equality is used. The second is a bit more interesting: * > 10 is internally rewritten into a closure like -> $x { $x > 10 } (through currying).

The sequence operator then does some magic comparison, depending on the type of the matcher. This comparison is called "smartmatching", and is a concept that reappears in many places in Perl 6. Examples:

    # after the 'when' keyword:
    given $age {
        when 100    { say "congratulations!"      }
        when * < 18 { say "You're not of age yet" }
    }

    # after 'where':
    subset Even of Int where * %% 2;

    # with an explicit smartmatch operator:
    if $input ~~ m/^\d+$/ {
        say "$input is an integer";
    }

    # arguments to grep(), first() etc.:
    my @even = @numbers.grep: Even;

On the right-hand side of the ~~ operator, and after when and where, the value to be matched against is set to the topic variable $_. This allows us to use constructs that operate on $_, like regexes created with m/.../ and .method_call.

Here is what the smart match operator does with some matchers on the right-hand side:

    # is it of type Str?
    $foo ~~ Str
    # is it equal to 6?
    $foo ~~ 6
    # or is it "bar"?
    $foo ~~ "bar"
    # does it match some pattern?
    $foo ~~ / \w+ '-' \d+ /
    # Is it between 15 and 25?
    $foo ~~ (15..25)
    # call a closure
    $foo ~~ -> $x { say 'ok' if 5 < $x < 25 }
    # is it an array of 6 elems in which every odd element is 1?
    $foo ~~ [1, *, 1, *, 1, *]

The full table for smart matching behavior can be found at
http://perlcabal.org/syn/S03.html#Smart_matching.

Notice how, unlike in Perl 5, in Perl 6 smartmatching is the only syntactic way to match a regex, it has no special operator. Also, while most smart matching cases return a Bool, matching against a regex returns a Match object – which behaves appropriately in boolean context too.

You have probably started wondering: a’right, that for built-in types, how do I use it in my own classes? You need to write a special ACCEPTS method for it. Say we have our good, old class Point:

    class Point {
        has $.x;
        has $.y;
        method ACCEPTS(Positional $p2) {
            return $.x == $p2[0] and $.y == $p2[1]
        }
    }

Everything clear? Let’s see how it works:

    my $a = Point.new(x => 7, y => 9);
    say [3, 5] ~~ $a; # Bool::False
    say (7, 9) ~~ $a; # Bool::True

Perl 6 can now do exactly what you mean, even with your own classes.

Day 9 – The module ecosystem

The Perl 6 module database on http://modules.perl6.org is certainly not CPAN yet, but there are still a number of things worth using, or at least knowing about. There’s no "standard" module installer for Perl 6, like there’s a cpan shell for Perl 5, but the most commonly used and most often working is neutro. It’s a simple script fetching, building, and installing modules from the ecosystem, resolving dependencies and checking if the tests are passing: not much more we need. Let’s see how to install something interesting with it, say JSON::Tiny, a JSON parser.

First, we need to get neutro. We will assume you use git to obtain it; notice that git is also obligatory to download modules (all of them live on github currently).

    git clone git://github.com/tadzik/neutro.git
    cd neutro
    PERL6LIB=tmplib bin/neutro .

That will download neutro and bootstrap it using the supplied libs. What we end up is the module installer itself, and the File::Tools and Module::Tools distributions. Make sure ~/.perl6/bin is in your PATH enviroment variable, so you will be able to run neutro without specifying its exact location. You are now able to install modules as simply as with cpanminus:

    neutro json
    neutro perl6-Term-ANSIColor
    neutro perl6-lwp-simple

You may notice module names are not similar to what you may be used to from Perl 5. They’re not standarized, they’re just the names of a git repos they live in. To make sure what you are looking for, consult the
list:

    neutro update # fetch the fresh list of modules
    neutro list

Modules will be installed to ~/.perl6/lib, which is in the default search path of Rakudo, so you don’t really need to set PERL6LIB yourself:

    perl6 -e 'use Term::ANSIColor; say colored("Hello blue world!", "blue")'

You probably just can’t wait to write your first module and make it available for the whole world. There’s no CPAN where you can send your packages; the usual workflow is creating a repository on Github and adding it to the projects.list file in the ecosystem. You don’t need to have a direct access to the repo to get your module published. You can either send a pull request for your forked ecosystem repo, send a patch, or just ask some of the commiters or people on the #perl6 channel of Freenode.

How to write a module, what tools to use? If you’re interested, look into the guide.

Happy hacking!

Day 1 – Reaching the Stars

Coming from the Perl 5 world, or from some other background, you may think of the programming language and its implementation as the one thing, or at least things very strongly tied to each other. Perl 6 is different. The “official” part is the specification (the Synopses) and the tests suite. Perl 6 encourages multiple implementations. Any implementation passing the official test suite and fulfiling the Synopses may call itself “a Perl 6 implementation”. While there is still no such implementation, there is alredy a few compilers in the ecosystem.

Rakudo is targetting the Parrot virtual machine, and it’s the most complete implementation so far. Niecza is targetting .NET and is aiming to study performance issues. There is also Yapsi, whose “author has claimed that it’s official for over half a year now, and not once been contradicted” :) We will be using Rakudo for our examples as it is the most complete implementation around and has the largest ecosystem built around it.

Like with Perl 6, there is no “one Rakudo to rule them all”. The Rakudo development team has decided to keep the compiler separate from the distribution. That means the Rakudo release is not a ready-to-use Perl 6 tarball, with modules, documentation and a virtual machine. That’s what Rakudo Star is. First released about half a year ago, Rakudo Star contains the toolbox with everything needed for Perl 6 hacking: a release of Parrot virtual machine, the Perl 6 Book, a bunch of modules and the Rakudo itself.

Getting Rakudo Star

The Rakudo Star tarballs are located at https://github.com/rakudo/star/downloads. Go get one suitable for your system and become ready for Christmas… again!