Day 20: Advancements in Macrotechnologies

Hi there!

Please allow me, on this day of the advent calendar, a small tangent. I’m not going to be talking directly about a cool Perl 6 feature of idiom. Instead, I’ll open a small window on what could be – and hopefully will be at some point!

If you, like me, have been following the progress on Rakudo for a few years, you have seen this bit regularly in the releases: > Some of the not-quite-there features include: > – advanced macros

Well, what does that exactly mean? Perl 6 does have macros, but they are currently limited beyond what people usually want to do. It’s not that they’re currently useless – they are still useful, from other posts in previous years of the advent showcasing them, to OO::Monitor using macros to report typos earlier.

Enter 007. 007 is a “Small experimental language with a license to macro”. What does this mean?! It’s a language that’s been created to temper and experiments with macros, so that their design is ready and battle-tested when they get integrated into Perl 6.

So, what’s in it? 007 tries to mimic the “powerful” parts of Perl 6, so that we don’t design macros for a completely different language. That means phasers, infix operators, (the gist of) a MOP and regexes are in.

What does it look like? At its core, 007 is meant to like Perl 6. It does, however, shed some parts of it. Let’s take a look at the most important snippet you’d want to write: FizzBuzz. Note: all code snippets in this blogpost are executable 007 code, not Perl 6 code.

my n = 1;
while n <= 100 {
    if n %% 15 {
        say("FizzBuzz");
    }
    else if n %% 3 {
        say("Fizz");
    }
    else if n %% 5 {
        say("Buzz");
    }
    else {
        say(n);
    }
    n = n + 1;
}

What? You don’t care about that? Well, obviously. I did promise you macros. We’re going to take a look at a simple macro, “name”, that returns the name of either the object of the last index.

macro name(expr) {
    if expr ~~ Q::Postfix::Property {
        expr = expr.property;
    }
    if expr !~~ Q::Identifier {
        throw new Exception {
            message: "Cannot turn a " ~ type(expr) ~ " into a name"
        };
    }
    return quasi { expr.name };
}
my info = {
    foo: "Bond",
    bar: {
        baz: "James Bond"
    },
};
say(name(info)); # info
say(name(info.foo)); # foo
say(name(info.bar.baz)); # baz

So, you’re probably going “WAT” right here. You’re right – this gist is missing some explanations. One of the most important feature of macro is access to the AST (Abstract Syntax Tree). Macros need to be able to mess with the code’s structure (like in Lisp), not the code text (like in C). The Q:: types are standardized types that represent the program’s shape. They do not particularly need to represent how the compiler/interpreter thinks about the code, but they need to be stable because we are writing our code – our macros – against this introspection API.

In this code sample, we use two Q types: Q::Postfix::Property, which represents the dot access, and Q::Identifier, which represents an identifier. First, we check if we have a property. If that’s the case, we extract what’s on the right side of the dot (remember, a.b.c is (a.b).c). We then check if we do end up with an identifier (and not, say, a number), and print that. This is e.g. how we could implement C#’s nameof operator, without having to add anything to the language!

A few days ago, masak++ published a blog post titled Has it been three years?, which marked the 3rd birthday of 007. While some areas are still (very) rough, it looks more and more like a usable language, and the possibilities progress day after day.

The next thing we are looking at is implementing is parsed. Here’s how it could look: (this example works in a PR, but uses special-casing right now):

macro statement:<whoa>() is parsed(/"whoa!"/) {
    return quasi @ Q::Statement {
        say("whoa!");
    }
};
whoa!;

This is also probably how we’d want them to look in Perl 6… Or not? The discussion is still open! You are encouraged to join the bikesh… discussion :-). The language is still young and needs a lot of fleshing out, as much for its advanced features as for its simpler features.

Before I leave you with your appriopriate amount of meta-fun, here’s a milestone 007 wants to reach that’s currently sitting in a branch: implement infix: as part of the library (if you’re unsure, the Perl 6 docs on ff apply here), and not as part of the language. Here goes the code!

# our infix macro takes a lhs (left hand side) and a rhs (right hand side).
macro infix:(lhs, rhs) is tighter(infix:<=>) {
    my active = False; # our current value when starting
    return quasi {
        if {{{lhs}}} {
            active = True; # if the bit on the left is true, we switch to active mode
        }
        my result = active; # the result we are returning
        if {{{rhs}}} {
            active = False; # if the bit on the right is true, we switch to inactive mode
        }
        result; # return the result stored *before* the rhs ran.
    };
}
my values = ["A", "B", "A", "B", "A"];
for values -> v {
    if v == "B" ff v == "B" {
        say(v);
    }
    else {
        say("x");
    }
}

That’s it for today! If you want more, feel free to check out the Tutorial, or the examples folder. If you want to learn about the will-be, we also have a Roadmap.

Day 19 – Introspection

Perl 6 is an Object-Oriented Programming Language. Well, really, it’s multi-paradigm; but one of the options (that’s used throughout its core) is definitily Object Orientation.

However, as the (current) perl6.org tells us, Perl 6 supports “generics, roles and multiple dispatch”, which are some very nice features, and already covered in other advent calendar posts.

But the one we’re going to take a look at today is the MOP. “MOP” stands for “Meta-Object Protocol”. It means that, instead of objects, classes, etc defining the language; they’re actually a part you can change (or just look at) from the user’s side of things.

Indeed, in Perl 6, you can add methods to a type, remove some, wrap methods, augment classes with more capabilities (OO::Actors and OO::Monitors are two such examples), or you could totally redefine it (and, say, use a Ruby-like object system. Such an example here).

But today, instead, we’re gonna look at the first part: Introspection. Looking at a type after it’s been built, getting to know it, and use these informations.

The module we’re going to build together is based on a need for the Sixcheck module (a QuickCheck-like module): generate some random data for a type, then feed that data to the function we’re testing, and check some post-condition.

So, we write our first version:

my %special-cases{Mu} =
  (Int) => -> { (1..50).pick },
  (Str) => -> { ('a'..'z').pick(50).join('') },
;
sub generate-data(Mu:U \t) {
  %special-cases{t} ?? %special-cases{t}() !! t.new;
}
generate-data(Int);

okay, that’s the first version we wrote. We note a few things:

  • We specify the key type for %special-cases. That’s because, by default, the type is Str. Obviously, we don’t want our types to be stringified. What we actually do is specify they’re going to be subtypes of “Mu” (which is at the top of the types “food chain”)
  • We put parentheses around Int and Str, to avoid stringification.
  • We use the :U in the function’s argument’s type. That means the value has to be undefined. Type objects (as in Int, Str, etc) are undefined, so it serves us well (a different unknown value you’ve probably seen is Nil).
  • Type objects really are… Objects, like any other object (more on that later).
    That’s why we can call .new on them, like here. It’s the same as directly calling Int.new, for example (that’s useful for consistency, and autovivification).
  • We provide a fallback for Int and Str, because calling Int.new and Str.new (0 and “”) would not give us any randomness in the data we create
  • Perl 6 automatically returns the last expression in a function, so we didn’t have to put a return there.

That’s our code to generate the data, fair and square. But we’re gonna need to generate much more than those simple examples.

The least we need to support is classes with properties: we want to look at the list of attributes, generate data for their type, and feed them to the constructor (the only classes we support right now are those with a parameterless constructor).

We need to be able to look inside a class. What we’re going to reach for, in Perl 6 terms, is the Meta-Object Protocol (or MOP for short). First off, let’s define a sample data class for our, huh, blog (the author is sorry for such a lack of imagination).

class Article {
  has Str $.title;
  has Str $.content;
  has Int $.view-count;
}
# we can manually create instances this way:
Article.new(title => "Perl 6 Advent, Day 19",
            content => "Magic!",
            view-count => 0);

But we don’t want to create the article by hand. We want to pass the class Article to our generate-data function, and get an Article back (with random data inside). Let’s go back to our REPL…

> say Article.^attributes;
(Str $!title Str $!content Int $!view-count)
> say Article.^attributes[0].WHAT;
(Attribute)

If you’ve clicked on the MOP link, you shouldn’t be surprised that we get a 3-elements array. If you’re still surprised by the syntax, .^ is the meta-method call. What it means is that a.^b translates to a.HOW.b(a).

If we want to know what’s available to us, we could just ask (removing the anonymous ones):

> Attribute.^methods.grep(*.name ne '<anon>')
(compose apply_handles get_value set_value container readonly package inlined WHY set_why Str gist)
> Attribute.^attributes
Method 'gist' not found for invocant of class 'BOOTSTRAPATTR'

Whoops… Seems like this is a bit too meta. Thankfully, we can use a very nice property of Rakudo: a lot of it is written in Perl 6! To know what’s available to us, we can just look up the source:

#     has str $!name;
...
#     has Mu $!type;

We got the name for the key, and the type to generate the value. Let’s see…

> say Article.^attributes.map(*.name)
($!title $!content $!view-count)
> say Article.^attributes.map(*.type)
((Str) (Str) (Int))

Yep! Seems correct! (If you’re wondering why we get $! (private) twigils back, it’s because $. only means a getter method will be generated. The attribute itself is still private, and accessible in the class).

Now, the only thing we need to build is a loop…

my %args;
for Article.^attributes -> $attr {
 %args{$attr.name.substr(2)} = generate-data($attr.type);
}
say %args.perl;

This is an example of what could be printed:

{:content("muenglhaxrvykfdjzopqbtwisc"), :title("rfpjndgohmasuwkyzebixqtvcl"), :view-count(45)}

You get different results each time you run your code (I don’t think it’ll produce an article worth reading, however…).

Only thing left to do is pass them to Article‘s constructor:

say Article.new(|%args);

(the prefix pipe | allows us to pass %args as named arguments, instead of a single positional argument). Again, you should get something like this printed:

Article.new(title => "kyvphxqmejtuicrbsnfoldgzaw", content => "jqbtcyovxlngpwikdszfmeuahr", view-count => 26)

Yay! We managed to create an Article instance “blindly”, without knowing nothing special about Article at all. Our code can be used to generate data for any constructor that expects its class attributes to be passed. Done!

Before I go, I need to remind you of one things: with great power comes a lot of “WTF”. If you’re only taking a peek like we do here, and gathering information, it should be all fine. But don’t go around and change the internals of Rakudo’s classes, because no one knows what’d happen then :-). As always, have a moderate amount to fun introspecting.

PS: Left as an exercise to the reader is the recursive implementation, move the loop to generate-data, so that we could add a User $.author attribute to Article, and get this one constructed as well. Good luck!

Day 18 – Sized, Typed, Shaped

If you’ve used Perl 5, you probably know PDL (Perl Data Language), the library to manipulate highly efficient data structures.

In Perl 6, we do not (yet) have full-fledged support for PDL (or a PDL-like module), but we do have support for specific features: sized types, shaped variables (and typed arrays).

The first feature, sized types, are low-level types that are composed of a generic low-level type name, and the number of bytes they use. Such an example is int1, int64 (aka int on 64-bit machines), buf8 (a “normal” byte buffer). If you’re interesting in reading the spec, it’s all in S09.

The second feature, shaped variables (still in S09), allow us to specify the fixed size of an array:

my @dwarves[7];

This means the only available indices are 0 to 6 (inclusive). This is an example from the Perl 6’s REPL (Read-Eval-Print-Loop. The input lines start with “>”):

> my @dwarves[7];
[(Any) (Any) (Any) (Any) (Any) (Any) (Any)]
> @dwarves[0] = 'Sleepy';
Sleepy
> @dwarves[7];
Index 7 for dimension 1 out of range (must be 0..6)

The last line of input mentions “dimension 1”. Indeed, Perl 6’s shaped variables can be multi-dimensional (see S09):

> my @ints[4;2];
[[(Any) (Any)] [(Any) (Any)] [(Any) (Any)] [(Any) (Any)]]
> @ints[4;3]
Index 3 for dimension 2 out of range (must be 0..1)
> @ints[4;1]
Index 4 for dimension 1 out of range (must be 0..3)
> @ints[0;0] = 1;
1
> @ints
[[1 (Any)] [(Any) (Any)] [(Any) (Any)] [(Any) (Any)]]

The first output shows us that Perl 6 filled our array with Any at first: this means the whole array is “empty”, and that it can store anything (since Any is a supertype of IntStr, etc).

We might not want that: our array should contain integers (Int). Instead, we can declare a typed array:

> my Int @a[8] = 0..7;
[0 1 2 3 4 5 6 7]
> @a[8] = 8;
Index 8 for dimension 1 out of range (must be 0..7)
> @a[0].WHAT.say # print the type
(Int)

This is all very useful in itself, but there’s a special property we can benefit from, by using everything presented in this post together: contiguous storage!

Natively-typed shaped arrays in Perl 6 are specced to take contiguous memory, meaning we get a very good memory layout, and use little memory

> my int @a[2;4] = 1..4, 5..8;
[[1 2 3 4] [5 6 7 8]]
> @a[0;0]++;
2
> @a.WHAT.say
(array[int])

Because we know how much memory each element takes, and how many elements we have, we can very easily calculate the array upfront, once and for all – or even calculate the size necessary to store the elements: we know that int, on our 64-bit computers (that’s int64), we need 2*4*8 bytes for our 2*4 array.

Wrapping up; that’s 64 bytes total (…almost), and we get to still write Perl 6! We can use the operations we know and love (even meta/hyper-operators) and al – they’re still arrays – with our cool performance boost. Amazing!

(note about the size: we need to add some overhead for the runtime; on MoarVM, that’s something like 16 bytes for GC header, 16 bytes for the dimensions list. That’s a grand total of 16+16+64=96 bytes. pretty cool!)

Day 22 – The Cool subset of MAIN.

In the Unix environment, many scripts take arguments and options from the command line. With Perl 6 it’s very easy to accept those. At least, that’s what Our 2010 advent post on MAIN says.

But today, we’re interested in going further, and we’re going to try to gain more expressiveness from it.
The first Perl 6 feature we’re going to look at here is subsets. Subsets have a very broad range of applications, but they’re even better when used in conjunction with MAIN.

Subsets are a declarative way to specify conditions on your type. If you wanted to translate “A QuiteSmallInt is an Int with a value lesser than 10”, you can write

subset QuiteSmallInt where * < 10;

In case you don’t remember from our 2010 advent post about smart matching or our 2011 advent post about multi-method dispatch, here’s a quick reminder of how you can use subsets with smart matching, multi-method dispatch or typed variables.

say 9 ~~ QuiteSmallInt; # True
say 11 ~~ QuiteSmallInt; # False

multi sub small-enough(QuiteSmallInt) { say "Yes!" }
multi sub small-enough($) { say "No, too big."; }

small-enough(9); # Prints "Yes!"
small-enough(11); # Prints "No, too big."

# You can also use it to type variables
my QuiteSmallInt $n = 9; # Works!
my QuiteSmallInt $n = 11; # Errors out with "Type check failed in assignment to '$n'; ..."

Alright; now that we got this out of the way, what does this buy us?
Well, as we just demonstrated, we can dispatch to different subroutines using those “where” conditions.
That goes for MAIN as well.

Say we wanted to take a filepath as the first argument of a MAIN. We’ll also write two “companions” that’ll give a descriptive error message in case we pass bad arguments.

# a File is a string containing an existing filename
subset File of Str where *.IO.e; # .IO returns an IO::Path object, and .e is for "exists"
subset NonFile of Str where !*.IO.e;

multi sub MAIN(File $move-from, NonFile $move-to) {
  rename $move-from, $move-to;
  say "Moved!";
}
multi sub MAIN($, File $) {
  say "The destination already exists";
}
multi sub MAIN(NonFile $, $) {
  say "The source file doesn't exist";
}

And now, if we try to use it (after saving it as “main.p6”)…

$ touch file1.txt
$ perl6 ./main.p6 file1.txt destination.txt
Moved!
$ perl6 ./main.p6 non-existant.p6 non-existant.txt
The source file doesn't exist!
$ perl6 ./main.p6 destination.txt destination.txt
The destination already exists

Alright, looks good. There’s a little catch, however: the –help will display 3 usages, when there’s only one.

$ perl6 ./main.p6 --help
Usage:
tmp.p6 <move-from> <move-to>
tmp.p6 <Any> (File)
tmp.p6 (NonFile) <Any>

Whoops! We need to find a way to hide them (if you’re wondering, the “<Any>” is because the arguments both are unnamed and untyped).
Luckily enough, Perl6 provides a trait for that: hidden-from-USAGE, added by moritz++ for this advent post (USAGE is the –help message).

Let’s add those to our MAINs.

multi sub MAIN(File $move-from, NonFile $move-to) {
  rename $move-from, $move-to;
  say "Moved!";
}
multi sub MAIN($, File $) is hidden-from-USAGE {
  say "The destination already exists";
}
multi sub MAIN(NonFile $, $) is hidden-from-USAGE {
  say "The source file doesn't exist";
}

And now, the USAGE is correct:

$ perl6 ./main.p6 --help
Usage:
tmp.p6 <move-from> <move-to>

Okay, now, we just need to add a description.

The second Perl6 feature we’re going to use is Pod.
Pod is an easy-to-use markup language, which you can embed to document your code.

#= Rename a file
multi sub MAIN(File $move-from, NonFile $move-to) {
  rename $move-from, $move-to;
  say "Moved!";
}

# You can write it in multiple lines, though in USAGE they'll be collapsed
# (joined by a space)

#={Rename
a
file}
multi sub MAIN(File $move-from, NonFile $move-to) {
  rename $move-from, $move-to;
  say "Moved!";
}

Both versions above will print the same –help:

$ perl6 ./main.p6 --help
Usage:
tmp.p6 <move-from> <move-to> -- Rename a file

And now you have everything you need to create a nice CLI interface :-).