Day 4 – New Perl 6 POD Features for the New Year

A tale of Santa’s helpers hacking the Rakudo compiler to fill a repo branch with POD goodies for the New Year.

Introduction

Rakudo NQP files contain the code that parses a Perl 6 input file and transforms it into a running Perl 6 program. This article will highlight some details learned by experience during recent work with Rakudo NQP files. The work involves implementing some not-yet-implemented (NYI) Perl 6 POD features, and I hope to merge the changes soon.

Preparation

The NQP files used are kept in a git repository at https://github.com/rakudo/rakudo/src/Perl6. See my 2017 Perl 6 Advent entry at https://perl6advent.wordpress.com/2017/12/08/ for more background on my development setup and work flow.

Background

During my work on implementing NYI POD features, I’ve added notes to a document I’ve added to the Rakudo repository: rakudo/docs/rakudo-nqp-and-pod-notes.md. I update it as I find out new things that may not be documented or whose documentation may not be easily found. That document also contains a complete list, by my reckoning, of NYI POD features. Following is a list of NYI POD features I’ve been working on for months which I expect to complete this year or early in the new year (along with roast tests for each):

  1. NYI: %config :numbered aliasing with ‘#’ for paragraph or delimited POD blocks
  2. NYI: POD data blocks
  3. NYI: formatting code in defn block terms

The missing items are described in the beautifully-crafted Synopsis S26 written by Dr. Damian Conway, Larry Wall’s prolific right-hand man-about-the-world Perl expert and renowned Perl author. (Note that few people are actively working on POD now and my total list of NYI features may be incomplete. S26 is very tightly written and not easy for me to understand without intense concentration. I spent quite a few wasted hours attempting to implement a feature that I thought was described but I misread the document!)

The work has taken me longer than I expected due to many factors which I will briefly discuss in the hope that it may help a future developer.

Rakudo NQP grammar and actions: lessons learned

Match objects

The completion of a grammar match on a token results in a match object. If the token has an action method with the same name as the token, that method is called with the match object as an implicit or explicit argument. By convention, ‘$/’ is used as the explict argument, but another name can be used (don’t do that!). I do not recommend ever relying on the implicit argument. Other arguments may be added if needed.

Note that, as the parse continues, match data is retained in the match object as it is used in other tokens and methods.

Assertions

Assertions are important in dynamic grammars found in POD handling. Several choices of match path often have to be made during a major match. One example, which gave me much trouble debugging incorrect use, was inside a token defining a delimited text block.

The test case that triggered the problem is file ‘b.t’:

=begin pod
text
=end pod

my $o = $=pod[0];
say $o;

When I ran perl6 against it I got

$ ./perl6 b.t
Preceding context expects a term, but found infix = instead.
Did you make a mistake in Pod syntax?
at /usr/local/people/tbrowde/mydata/tbrowde-home-bzr/perl6/perl6-repo-forks/rakudo/b.t:1
------> =begin ⏏pod

Not very helpful! Then I tried:

$ ./perl6 b.t --ll-exception b.t
Preceding context expects a term, but found infix = instead.
Did you make a mistake in Pod syntax?
   at SETTING::src/core/Exception.pm6:57  (./CORE.setting.moarvm:throw)
 from src/Perl6/World.nqp:4955  (blib/Perl6/World.moarvm:throw)
 from gen/moar/Perl6-Grammar.nqp:301  (blib/Perl6/Grammar.moarvm:typed_panic)
 from gen/moar/Perl6-Grammar.nqp:3609  (blib/Perl6/Grammar.moarvm:)
 ...more files and line numbers...

Even less useful! I tried manually investigating the listed files and couldn’t decipher the code well enough to get a clue.

Then I tried another, similar test case that appeared to work, file ‘b2.t’:

=begin table
text
=end table

my $o = $=pod[0];
say $o;

When I ran perl6 against it I got

$ ./perl6 b2.t
Pod::Block::Table
  text

Success!

But that failing test case caused me weeks of trying various debugging techniques until finally, after looking at the delimited token in Grammar.nqp one more time and mentally evaluating what each of its submatch groups was doing. I then looked more closely at this group containing an assertion:

[
    # defn-line is all text to the newline
     # <== assertion: this is a 'defn' type
    \s* 
]

That group, which is sequential and not part of an alternation, in the delimited block token definition must match or the total token fails. Unfortunately, the failure result was an exception that was LTA for the situation (which is not uncommon in NQP and one of the hazards of working in it), and I blundered around too long trying to find the cause. One of the things that fooled me was thinking that an assertion in a group which was not met was like a ‘?’ quantifier meaning that a failed match was ignored. After I studied it more carefully, I decided that is definitely not the case! The group either matches or not, so the quantifier must be there if a non-match is acceptable.

When I compared code for the delimited block token against the working code for the delimited_table block token (which I had done many times before), I saw that the same match group in the delimited_table block had the ‘?’ quantifier. After I added the ‘?’ to the group in the delimited block token, the bad test case worked again!

Debugging

The most useful grammar and actions debugging technique for me has been the classic one: print statements to show the value of variables during execution. The method varies depending on which file type and what values are desired to be shown. Following are some examples:

  1. Showing a match object’s contents:
method do-foo($/) {
    say("DEBUG: dumping method 'do-foo' match:");
    say($/.dump);
}

  1. Showing results during a grammar match:
token blah {
    \h* $ = [ foo | bar ] # <== note '=' instead of ':='
    { say("DEBUG: \$ value: '{$}'"); }
}

Notice the say statement is inside a block defined by curly braces. Notice also, even in an NQP source file, the assignment operator (‘=’) for match objects used in a grammar instead of the binding operator (‘:=’).

Dynamic variables

Grammar and actions make heavy use of dynamic variables (variables with a * twigil, e.g., $*IN-DEFN-BLOCK). They show their versatility when a variable needs to be changed deep in a parse tree and that value is expected to hold during the remainder of that parse (the caller) and subparse actions.

make, made, and ast

The terms make, made, and ast, as used in grammar and actions, have always confused me in spite of explanations in all the published Perl 6 books. Thanks to further explanation and answers to my questions on IRC #perl6-dev by Perl 6 author Moritz Lenz, they are much clearer.

Basically, inside an action method, using make will assign the current value to the .ast attribute (or its alias .made) of the match object with the method’s name. So, given the following method:

method do-foo($/) {
    my $val = 6;
    make $val;
}

or, alternatively:

method do-foo($/) {
    $/.ast := 6;
}

We later can get that value back with one of these idioms:

say("do-foo.ast = {$.ast}");  # output: 6
say("do-foo.ast = {$.made}"); # output: 6

The choice of the attribute name .ast is misleading since it commonly refers to abstract syntax tree (AST) but, in this case, it has nothing to do with an AST (although it may have as its value a QAST node or any other type of NQP object).

Note that any value assigned to an .ast attribute may be overwritten or removed at a later stage in the grammar or actions.

Deferring generation of QAST nodes

Sometimes premature generation of QAST nodes in existing grammar has prevented proper POD feature implementation. An example is the %config section of a POD block which has some values needed for later parsing. Part of the work I’m doing requires reworking %config match code so the QAST node is not generated until all parts of the parent object (usually a POD class) have been evaluated or constructed as needed.

Summary

I have gradually found out how to reform the Rakudo Perl 6 grammar and actions to implement some NYI POD features, and I expect to deliver them soon. During the work, I have learned many lessons the hard way and hope I have shed some light on the darker corners of POD parsing.

One final lesson to take away from any major coding project: make, test, and commit small (i.e., limited) changes for a merge commit! I got so wrapped up in the sometimes-crooked parse path of POD features that I made too many changes and couldn’t easily undo them. I hope I don’t repeat that mistake.

I wish for you and yours a Perl 6ish Merry Christmas and Happy New Year, and, in the immortal words of Charles Dickens’s Tiny Tim (A Christmas Carol), may “God bless Us, Every One!”