Author Archive

Grammar::Tracer and Grammar::Debugger

December 2, 2011

Grammars are, for many people, one of the most exciting features of Perl 6. They unify parsing with object orientation, with each production rule in your grammar being represented by a method. These methods are a little special: they are declared using the keywords “regex”, “rule” or “token”, each of which gives you different defaults on backtracking and whitespace handling. In common is that they lead to the body of the method being parsed using the Perl 6 rule syntax. Under the hood, however, they really are just methods, and production rules that refer to others are really just method calls.

Perl 6 grammars also give you a seamless way to combine declarative and imperative parsing. This means efficient mechanisms, such as NFAs and DFAs, may be used to handle the declarative parts – the things that your tokens tend to be made up of – while a more imperative mechanism drives the parsing of larger structures. This in turn means that you don’t need to write a tokenizer; it can be derived from the rules that you write in the grammar.

So what is the result of parsing some text with a grammar? Well, provided it’s able to match your input, you get back a parse tree. This data structure – made up of Match objects – captures the structure of the input. You can treat each Match node a little bit like a hash, indexing in to it to look at the values that its production rules matched. While you can build up your own tree or other data structure while parsing, sometimes the Match tree you get back by default will be convenient enough to extract the information you need.

That’s wonderful, but there was a key clause in all of this: “provided it’s able to match”. In the case that the grammar fails to match your input, then it tells you so – by giving back an empty Match object that, in boolean context, is false. It’s at this point that many people stop feeling the wonder of grammars and start feeling the pain of trying to figure out why on earth their seemingly fine grammar did not accept the input they gave it. Often, it’s something silly – but in a grammar of dozens of production rules – or sometimes even just ten – the place where things go wrong can be elusive.

Thankfully, help is now at hand, in the form of two modules: Grammar::Tracer, which gives you a tree-like trace output of your grammar, and Grammar::Debugger, which gives the same trace output but also enables you to set breakpoints and single step through the grammar.

A picture is worth a thousand words, so here’s how Grammar::Tracer looks in action!

What we’re seeing here is a tree representation of the production rules that were called, starting at “TOP”, next trying to parse a production rule called “country”, which in turn wants to parse a name, two “num”s and an “integer”. The green indicates a successful match, and next to it we see the snippet of text that was captured.

So what happens when things go wrong? In that case, we see something like this:

Here, we see that something happened during the parse that caused a cascade of failures all the way back up to the “TOP” production rule, which meant that the parse failed overall. Happily, though, we now have a really good clue where to look. Here is the text my grammar was trying to match at the time:

	Ulan Ude : 51.833333,107.600000 : 1
	Moscow : 55.75000,37.616667 : 4

Looking at this, we see that the “name” rule appears to have picked up “Ulan”, but actually the place in question is “Ulan Ude”. This leads us directly to the name production in our grammar:

token name { \w+ }

Just a smattering of regex fu is enough to spot the problem here: we don’t parse names that happen to have spaces in them. Happily, that’s an easy fix.

token name { \w+ [\h+ \w+]* }

So how do we turn on the tracing? Actually, that’s easy: just take the file containing the grammar you wish to trace, and add at the top:

use Grammar::Tracer;

And that’s it; now whenever you use the grammar, it will be traced. Note that this statement has lexical effect, so if you’re using modules that also happen to have grammars – which you likely don’t care about – they will not end up getting the tracing behavior.

You can also do this:

use Grammar::Debugger;

The debugger is the tracer’s big sister, and knows a few more tricks. Here’s an example of it in action.

Instead of getting the full trace, now as soon as we hit the TOP production rule the program execution breaks and we get a prompt. Pressing enter allows you to step rule by rule through the parse. For some people, this may be preferable; others prefer to get the full trace output and analyze it. However, there are a few more tricks. In the example above, I added a breakpoint on the “name” rule. Using “r” informs the debugger to keep running through the production rules until it hits one called “name”, at which point it breaks. It is also possible to add breakpoints in code, for more extended debugging sessions with many runs. There’s one additional feature in code, which is to set a conditional breakpoint.

Sound interesting? You can get modules from GitHub, and if you want to see a live demo of a grammar being debugged using it, then there is a video of my Debugging Perl 6 Grammars talk from YAPC::Europe 2011; slides are also available to make the sample code more clear than it is on the video. Note that the modules need one of the compiler releases from the Rakudo “nom” development branch; we’ll be making a distribution release later this month based on that, though, and these modules will come with it.

You may also be thinking: I bet these are complex modules doing lots of guts stuff! In fact, they are 44 lines (Grammar::Tracer) and 171 lines (Grammar::Debugger), and written in Perl 6. They are built using the meta-programming support we’ve been working on in the Rakudo Perl 6 compiler during the course of the last year – and if you want to know more about that, be sure to check out my meta-programming post coming up later on in this year’s advent calendar.

Day 18: Roles

December 18, 2009

As the snow falls outside, we grab a glass of mulled wine – or maybe a cup of eggnog – to enjoy as we explore today’s exciting gift – roles!

Traditionally in object oriented programming, classes have taken on two tasks: instance management and re-use. Unfortunately, this can end up pulling classes in two directions: re-use wants them to be small and minimal, but if they’re representing a complex entity then they need to support all of the bits it needs. In Perl 6, classes retain the task of instance management. Re-use falls to roles.

So what does a role look like? Imagine that we are building up a bunch of classes that represent different types of product. Some of them will have various bits of data and functionality in common. For example, we may have a BatteryPower role.

role BatteryPower {
    has $.battery-type;
    has $.batteries-included;
    method find-power-accessories() {
        return ProductSearch::find($.battery-type);

At first glance, this looks a lot like a class: it has attributes and methods. However, we can not use a role on its own. Instead, we must compose it into a class, using the does keyword.

class ElectricCar does BatteryPower {
    has $.manufacturer;
    has $.model;

Composition takes the attributes and methods – including generated accessors – from the role and copies them into the class. From that point on, it is as if the attributes and methods had been declared in the class itself. Unlike with inheritance, where the parents are looked at during method dispatch, with roles there is no runtime link beyond the class knowing to say “yes” if asked if it does a particular role.

Where things get really interesting is when we start to compose multiple roles into the class. Suppose that we have another role, SocketPower.

role SocketPower {
    has $.adapter-type;
    has $.min-voltage;
    has $.max-voltage; 
    method find-power-accessories() {
        return ProductSearch::find($.adapter-type);

Our laptop computer can be plugged in to the socket or battery powered, so we decide to compose in both roles.

class Laptop does BatteryPower does SocketPower {

We try to run this and…BOOM! Compile time fail! Unlike with inheritance and mix-ins, role composition puts all of the roles on a level playing field. If both provide a method of the same name – in this case, find-power-accessories – then the conflict will be detected as the class is being formed and you will be asked to resolve it. This can be done by supplying a method in our class that says what should be done.

class Laptop does BatteryPower does SocketPower {
    method find-power-accessories() {
        my $ss = $.adapter-type ~ ' OR ' ~ $.battery-type;
        return ProductSearch::find($ss);

This is perhaps the most typical use of roles, but not the only one. Roles can also be taken and mixed in to an object (on a per-object basis, not a per-class basis) using the does and but operators, and if filled only with stub methods will act like interfaces in Java and C#. I won’t talk any more about those in this post, though: instead, I want to show you how roles are also Perl 6’s way of achieving generic programming, or parametric polymorphism.

Roles can also take parameters, which may be types or just values. For example, we may have a role that we apply to products that need to having a delivery cost calculated. However, we want to be able to provide alternative shipping calculation models, so we take a class that can handle the delivery calculation as a parameter to the role.

role DeliveryCalculation[::Calculator] {
    has $.mass;
    has $.dimensions;
    method calculate($destination) {
        my $calc =
        return $$destination);

Here, the ::Calculator in the square brackets after the role name indicates that we want to capture a type object and associate it with the name Calculator within the body of the role. We can then use that type object to call .new on it. Supposing we had written classes that did shipping calculations, such as ByDimension and ByMass, we could then write:

class Furniture does DeliveryCalculation[ByDimension] {
class HeavyWater does DeliveryCalculation[ByMass] {

In fact, when you declare a role with parameters, what goes in the square brackets is just a signature, and when you use a role what goes in the square brackets is just an argument list. Therefore you have the full power of Perl 6 signatures at your disposal. On top of that, roles are “multi” by default, so you can declare multiple roles with the same short name, but taking different types or numbers of parameters.

As well as being able to parametrize roles using the square bracket syntax, it is also possible to use the of keyword if each role takes just one parameter. Therefore, with these declarations:

role Cup[::Contents] { }
role Glass[::Contents] { }
class EggNog { }
class MulledWine { }

We may now write the following:

my Cup of EggNog $mug = get_eggnog();
my Glass of MulledWine $glass = get_wine();

You can even stack these up.

role Tray[::ItemType] { }
my Tray of Glass of MulledWine $valuable;

The last of these is just a more readable way of saying Tray[Glass[MulledWine]]. Cheers!

Day 11: Classes, attributes, methods and more

December 11, 2009

We excitedly tear the shiny wrapping paper off today’s gift, and inside we find something that nobody could object to! It’s the Perl 6 object model, in all its class-declaring, role-composing, meta-modelling glory. But before we get too carried away with the high-powered stuff, let’s see just how easy it is to write a class in Perl 6.

class Dog {
    has $.name;
    method bark($times) {
        say "w00f! " x $times;

We start off by using the class keyword. If you’re coming from a Perl 5 background, you can think of class as being a bit like a variant of package that gives you a bunch of classy semantics out of the box.

Next, we use the has keyword to declare an attribute along with an accessor method. The . that you see in the name is a twigil. Twigils tell you something special about the scoping of a variable. The . twigil means “attribute + accessor”. Other options are:

has $!name;       # Private; only visible in the class
has $.name is rw; # Generates an l-value accessor

Next comes a method, introduced using the method keyword. method is like sub, but adds an entry to the methods table of the class. It also automatically takes the invocant for you, so you don’t have to write it in the parameter list. It is available through self.

All classes inherit a default constructor, named new, which maps named parameters to attributes. We can call this on Dog – the type object of the Dog class – to get a new instance.

my $fido = => 'Fido');
say $;  # Fido
$fido.bark(3);   # w00f! w00f! w00f!

Notice that the method call operator in Perl 6 is . rather than Perl 5’s ->. It’s 50% shorter, and will be familiar to developers coming from a range of other languages.

Of course, there’s inheritance, so we can introduce a yappy puppy.

class Puppy is Dog {
    method bark($times) {
        say "yap! " x $times;

There’s also support for delegation.

class DogWalker {
    has $.name;
    has Dog $.dog handles (dog_name => 'name');
my $bob = => 'Bob', dog => $fido);
say $;      # Bob
say $bob.dog_name;  # Fido

Here, we declare that we’d like calls to the method dog_name on the class DogWalker to forward to the name method of the contained Dog. Renaming is just one option that is available; the delegation syntax offers many other alternatives.

The beauty is more than skin deep, however. Beneath all of the neat syntax is a meta-model. Classes, attributes and methods are all first class in Perl 6, and are represented by meta-objects. We can use these to introspect objects at runtime.

for Dog.^methods(:local) -> $meth {
    say "Dog has a method " ~ $;

The .^ operator is a variant on the . operator, but instead makes a call on the metaclass – the object that represents the class. Here, we ask it to give us a list of the methods defined within that class (we use :local to exclude those inherited from parent classes). This doesn’t just give us a list of names, but instead a list of Method objects. We could actually invoke the method using this object, but in this case we’ve just ask for its name.

Those of you into meta-programming and looking forward to extending the Perl 6 object model will be happy to know that there’s also a declarational aspect to all of this, so uses of the method keyword actually compile down to calls to add_method on the meta-class. Perl 6 not only offers you a powerful object model out of the box, but also provides opportunities for it to grow to meet other future needs that we didn’t think of yet.

These are just a handful of the great things that the Perl 6 object model has to offer; maybe we’ll discover more of them in some of the other gifts under the tree. :-)


Get every new post delivered to your Inbox.

Join 44 other followers