Day 15 – Building a (little) Spacecraft with Perl 6

Showing off long ears

In the previous post, we have encountered some kind of special elves’ magic:

enum Fuel <Solid Liquid Gas>;

class SpeedChoice does ASNChoice {
    method ASN-choice { { mph => (1 => Int), kmph => (0 => Int) } }
}

class Rocket does ASNSequence {
    has Str $.name is UTF8String;
    has Str $.message is default-value("Hello World") is UTF8String;
    has Fuel $.fuel;
    has SpeedChoice $.speed is optional;
    has Str @.payload is UTF8String;

    method ASN-order { <$!name $!message $!fuel $!speed @!payload> }
}

my $rocket = Rocket.new(
    name => 'Falcon',
    fuel => Solid,
    speed => SpeedChoice.new((mph => 18000)),
    payload => [ "Car", "GPS" ]);

my $rocket-bytes = ASN::Serializer.serialize($rocket, :mode(Implicit));

#`[ Result:
      Blob.new(
          0x30, 0x1B, # Outermost SEQUENCE
          0x0C, 0x06, 0x46, 0x61, 0x6C, 0x63, 0x6F, 0x6E, # NAME, MESSAGE is missing
          0x0A, 0x01, 0x00, # ENUMERATED
          0x81, 0x02, 0x46, 0x50, # CHOICE
          0x30, 0x0A, # SEQUENCE OF UTF8String
              0x0C, 0x03, 0x43, 0x61, 0x72,  # UTF8String
              0x0C, 0x03, 0x47, 0x50, 0x53); # UTF8String
]

say ASN::Parser.new(:type(Rocket)).parse($rocket-bytes) eqv $rocket; # Certainly true!

However, as an elf I know once quoted, ‘It’s hard to tell the difference between mastered technique and magic’. So the mystery can be resolved? Let’s glance over the way it all works.

Types, types, types

Some things are pretty self-evident (or it seems so to me, after countless hours of looking at how elves do tricks):

# 1
role ASNSequence {
    # every descendant has to fulfill this important vow!
    method ASN-order {...}
}

# 2
role ASNChoice {
    has $.choice-value;

    # if you have to choose, choose wisely!
    method ASN-choice() {...}
    method ASN-value() { $!choice-value }

    method new($choice-value) { $?CLASS.bless(:$choice-value) }
}

# 3
role ASN::StringWrapper {
    has Str $.value;

    # Don't do this at home. :]
    method new(Str $value) { self.bless(:$value) }
}

# UTF8String wrapper
role ASN::Types::UTF8String does ASN::StringWrapper {}

# Yes, it is _this_ short
multi trait_mod:<is>(Attribute $attr, :$UTF8String) is export { $attr does ASN::Types::UTF8String }

• First one is a simple role which allows us to enforce ASN-order method presence.
• Second one is a role that holds an actual value of a CHOICE and enforces a method where user has to describe possible options.
• Third one describes a trait like is UTF8String that adds a role to an attribute, which will help us later, and provides the role itself along with some wrapper code.

In the same way as third piece is expressing, OPTIONAL, DEFAULT “traits” and other string types can be expressed.

Advance, evolve, serialize!

What can one do to serialize a thing by a set of rules? Given that Basic Encoding Rules have different treating for values of different types (which isn’t too odd if you think about it!) and a fact that a type can be nested in another one, let alone be recursive? I have a feeling that it might be not too hard to implement. Perl 6’s multi-dispatch comes in handy!

Generally, things be like:

class ASN::Serializer {
    ...

    # like this:
    multi method serialize(ASNSequence $sequence, Int $index = 48, :$debug, :$mode = Implicit) { ... }

    # or this:
    multi method serialize(Int $int is copy where $int.HOW ~~ Metamodel::ClassHOW, Int $index = 2, :$debug, :$mode) { ... }
    multi method serialize($enum-value where $enum-value.HOW ~~ Metamodel::EnumHOW, Int $index = 10, :$debug, :$mode) { ... }

    # or even this:
    multi method serialize(Positional $sequence, Int $index is copy = 16, :$debug, :$mode) { ... }

    ...

The rules describing everything in this area are:

• For complex types one has to introduce, like ASNStructure, iterate over its content, serializing insides one by one, and join it properly. At the end of the day, for every piece such Serializer has an attribute type known or can infer it based on roles applied by traits (handy!), one can have attribute’s value (or can skip the attribute if it is optional and omitted), can wrap/unwrap Str-based types – all of this allows one to serialize a type.
• For simple types one can just serialize it according to rules given.
• For some handy “special cases”, for instance, @.foo-like attributes, one needs to infer what is happening (in this case, it is going to be a SEQUENCEOF type) and serialize it appropriately.

Besides first parameter with a value, there are three more:

• $index integer comes in handy, especially for BER specific indexing
• $debug flag enables debug output (which is very helpful when one is debugging some binary protocol!)
• $mode value might be used in future to support tagging schemas other than IMPLICIT.

If there is time to encode, there is always time to decode

What is a parser? If serializer is a “parser backward”, then a parser is a… yes, it is a serializer backward! But what does it mean? Generally, a serializer takes some A and produces some B of given form. And a parser takes some B of given form and produces some A.

Let’s say one knows the exact type which is being parsed:

my $parser = ASN::Parser.new(type => Rocket);
say $parser.parse($rocket-ber); # Yes, here goes our rocket!

If one is going to parse this Buf content, its type has to be specified, like this:

multi method parse(Blob $input, ...) {
    ...
    self.parse($input, $!type, ...);
}

This method does not know what type it parses, but it calls its friend: parse($input, SomeCoolType, ...) out of content passed and out of type it can get. And if one knows a type, multi-dispatch will gladly give us necessary parsing implementation. For a simple type. For a complex type. For a “special” type. With Perl 6 handy miracles happen any day!

Let’s glance a bit more over this:

# Details and basic indentation are omitted for clarity

...

multi method parse(Buf $input is rw, ASNSequence $type, :$debug, :$mode) {
    # `$type` here is, really, not a value, but a Type Object. As `ASN-order` is defined on
    # type, there are no problems with gathering necessary info:
    my @params = do gather {
        for $type.ASN-order.kv -> $i, $field {
            # Here be dragons! Or, rather, MOP is used here!
        }
    }
    # A-a-and a ready object of a type our parser has no clue about is returned.
    # Yes, it is kind of neat. :)
    $type.bless(|Map.new(@params));
}

Simpler types are, in fact, a lot simpler this that one, like this:

multi method parse(Buf $input is rw, $enum-type where $enum-type.HOW ~~ Metamodel::EnumHOW, :$debug, :$mode) {
    say "Parsing `$input[0]` out of $input.perl()" if $debug;
    $enum-type($input[0]);
}

However, one has to keep rules being kept, to indicate errors, and do all kind of “boring”, but “necessary” things. While Perl 6 allows us to use some nice tricks in this area too, it is not too much of interest to look there before Christmas.

What o’clock? Supply o’clock!

In case you are already tired from all this ASN.1 related stuff, I have a good news: it is nearly over. \o/

While all those “types are my first-class citizens and I am cool with it” tricks are fun, there is one more trick to show, while being related, yet kind of completely different.

ASN.1 parser should be incremental. What is more, it is quite clearly specified it must be, as one can work with values of yet unknown length. What can be done to quickly make our parser incremental? Let’s do it quickly:

class ASN::Parser::Async {
    has Supplier::Preserving $!out = Supplier::Preserving.new;
    has Supply $!values = $!out.Supply;
    has Buf $!buffer = Buf.new;
    has ASN::Parser $!parser = ASN::Parser.new(type => $!type);
    has $.type;

    method values(--> Supply) {
        $!values;
    }

    method process(Buf $chunk) {
        $!buffer.append: $chunk;
        loop {
            # Minimal message length
            last if $!buffer.elems < 2;
            # Message is incomplete, good luck another time
            last unless $!parser.is-complete($!buffer);
            # Cut off tag, we know what it is already in this specific case
            $!parser.get-tag($!buffer);
            my $length = $!parser.get-length($!buffer);
            # Tag and length are already cut down here, take only value
            my $item-octets = $!buffer.subbuf(0, $length);
            $!out.emit: $!parser.parse($item-octets, :!to-chop); # `!to-chop`, because "prefix" is already cut
            $!buffer .= subbuf($length);
        }
    }

    method close() {
        $!out.done;
    }
}

It can be used like this:

my $parser = ASN::Parser::Async.new(type => Rocket);

$parser.values.tap({ say "I get a nice thing!"; });

react {
    whenever $socket.data-arrived -> $chunk {
        $parser.process($chunk);
        LAST { $parser.close; }
    }
}

This is all one had to add to make such Parser incremental for this minimal case.

Of course, as you can guess, things I am writing about are a bit too specific to be just my imagination running wild with not only elves but a full party of adventurers (who can handle some binary stuff too!). The implementation is already available at ASN::BER repository. While it may be a very early alpha release, with a lot of stuff not even planned yet, and there are great lengths one can go about to improve overall state of this module, it is already being helpful for me to work on LDAP stuff I mentioned earlier semi-secretly. The repository is surely opened for suggestions, bug reports (and maybe even hug reports), as there are tons of work that still has to be done, but this is another story.

Have a good day and make sure to rest well during Christmas holidays!