Day 8 – Adventures in NQP Land: Hacking the Rakudo Compiler

With apologies to the old Christmas classic, “The Twelve Days of Christmas,” I give you the first line of a Perl 6 version:

On the first day of Christmas, my true love gave to me, a Perl table in a pod tree…

But the table I got wasn’t very pretty!

Background

My first real contact with Perl 6 came in the spring of 2015 when I decided to check on its status and found it was ready for prime time. After getting some experience with the language I started contributing to the docs in places where I could help. One of my first contributions to the docs was to clean up one of the tables which was not rendering nicely . During my experiments with pod tables on my local host I tried the following table:

=begin table
-r0c0 r0c1
=end table

which caused Perl 6 to throw an ugly, LTA (less than awesome) exception message:

"===SORRY!=== Cannot iterate object with P6opaque representation"

I worked around the problem but it nagged at me so I started investigating the guts of pod and tables. That led me to the source of the problem in github.com/rakudo/src/Perl6/Pod.nqp.

In fact, the real problem for many pod table issues turned out to be in that file.

Not Quite Perl (NQP)

nqp is an intermediate language used to build the Rakudo Perl 6 compiler. Its repository is found here. The rest of this article is about modifying nqp code in the rakudo compiler found in its repository here. Rakudo also has a website here.

Before getting too far I first read the available information about Rakudo and NQP here:

Then I started practicing nqp coding by writing and running small nqp files like this (file “hello.nqp”):

say("Hello, world!");

which, when executed, gave the expected results:

$ nqp hello.nqp
Hello, world!

Note that say() is one of the few nqp opcodes that doesn’t require the nqp:: prefix.

Into the trenches

The purpose of the Perl6::Pod class, contained in the rakudo/src/Perl6/Pod.nqp file, is to take pod grammar matches and transform them into Perl 6 pod class definitions, in rakudo/src/core/Pod.pm, for further handling by renderers in Perl 6 land. For tables that means anything represented in any legal pod form as described by the Perl 6 documentation design Synopsis S26, the Perl 6 test suite specs, and the Perl 6 docs has to be transformed into a Perl 6 Pod::Block::Table class object with this form as described in file rakudo/src/core/Pod.pm:

configuration information
a header line with N cells
M content lines, each with N cells

I wanted the nqp table pod handling to be robust and able to automatically fix some format issues (with a warning to the author) or throw an exception (gracefully) with detailed information of problems to enable the author to fix the pod input.

Workspace and tools

I needed two cloned repositories: rakudo and roast. I also needed forks of those same repositories on github so I could create pull requests (PRs) for my changes. I found a very handy Perl 5 tool in CPAN module App::GitGot. Using got allowed me to easily set up all four repos. (Note that got requires that its target repo not exist either in the desired local directory or the user’s github account.) After configuring got I went to a suitable directory to contain both repos and executed the following:

got fork https://github.com/rakudo/rakudo.git
got fork https://github.com/perl6/roast.git

which resulted in a subdirectories rakudo and roast containing the cloned repos and new forks of rakudo and roast on my github account. In the rakudo directory one can see the default setup for easy creation of PRs:

$ git remote -v
origin git@github.com:tbrowder/rakudo.git (fetch)
origin git@github.com:tbrowder/rakudo.git (push)
upstream https://github.com/rakudo/rakudo.git (fetch)
upstream https://github.com/rakudo/rakudo.git (push)

There are similar results in the roast repo.

Next, I renamed the roast repo as a subdirectory of rakudo (“rakudo/t/spec”) so it functions as a subgit of the local rakudo.

Finally, I created several bash scripts to ease configuring rakudo for installation in the local repo directory, setting the environment, and running tests:

  • rakudo-local-config.sh
  • run-table-tests.sh
  • set-rakudo-envvars.sh

(See all scripts mentioned here at https://github.com/tbrowder/nqp-tools.)

To complete the local working environment you will need to install some local modules so you must change your path and install a local copy of the zef installer. Follow these steps in your rakudo directory (from advice from @Zoffix):

git clone https://github.com/ugexe/zef
export PATH=`pwd`/install/bin:$PATH
cd zef; perl6 -Ilib bin/zef install .
cd ..
export PATH=`pwd`/install/share/perl6/site/bin:$PATH
zef install Inline::Perl5

Then install any other module you need, e.g.,

zef install Debugger::UI::CommandLine
zef install Grammar::Debugger

Hacking

Now start hacking away. When ready for a build, execute:

make
make install

The make install step is critical because otherwise, with the local environment we set up, the new Perl 6 executables won’t be found.

Debugging for me was laborious, with rebuilds taking about three minutes each. The debugger (perl6-debug-m) would have been very useful but I could not install the required Debbugger::UI::CommandLine module so it would be recognized by the locally-installed perl6-debug-m. The primary method I used was inserted print statements plus using the --ll-exception option of perl6. Of major note, though, is that this author is a Perl 6 newbie and made lots of mistakes, and did not always remember the fixes, hence this article. (Note I likely would have used the debugging tools but, at the time I started, I did not ask for help and did not have the advice provided shown above.)

Testing

It goes without saying that a good PR will include tests for the changes. I always create a roast branch with the same name as my rakudo branch. Then I submit both PRs and I refer to the roast PR in the rakudo PR and vice versa. I note for the roast PR that it requires the companion rakudo PR for it to pass all tests.

See Ref. 5 for much more detail on specialized test scripts for fudging and other esoteric testing matters.

Documentation

I try to keep the Perl 6 pod table documentation current with my fixes.

NQP lessons learned

  • LTA error messages are a fact of life, e.g., “Cannot invoke this object…”, which can be caused by many things, including a misspelled identifier (NQP issue filed, early report is it may be impossible to fix anytime soon).

  • Ensure all nqp opcodes have the nqp:: prefix (except the few built-ins)

  • Practice with new code in an nqp-only sand-box.

Success!

I have now had two major Perl 6 POD (and roast) PRs accepted and merged, and am working on one more “easy” one which I should finish this week. The PRs are:

  1. Rakudo PR #1240

The Rakudo PR provided fixes for RTs #124403, #128221, #132341, #13248, and #129862. It was accompanied by roast PR #353.

The PR allowed the problem table above to be rendered properly. It also added warnings for questionable tables, added Rakudo environment variables RAKUDO_POD6_TABLE_DEBUG to aid users in debugging tables (see docs, User Debugging), and allows short rows with empty columns to be rendered properly.

  1. Rakudo PR #1287

The Rakudo PR provided a fix for Rakudo repo issue #1282. It was accompanied by roast PR #361. (Note that roast PR #361 is not yet merged.)

The PR allows table visual column separators (‘|’) and (‘+’) to be used as cell data by escaping them in the pod source.

Summary

  • Perl 6 pod is a great improvement over Perl 5, but it is still not fully implemented.

  • Working in the bowels of Rakudo Perl is rewarding (and fun), but prepare to get your hands dirty!

  • The Perl 6 community is a great group to be associated with.

  • I love Rakudo Perl 6.

Merry Christmas and Happy Hacking!

Credits

Any successful inputs I have made are due to all the Perl 6 core developers and the friendly folks on IRC (#perl6, #perl6-dev).

References

  1. JWs Perl 6 debugger Advent article
  2. JWs Rakudo debugger module Debugger::UI::CommandLine
  3. JWs grammar debugger module Grammar::Debugger
  4. Testing Rakudo
  5. Contributing to roast
  6. Github guide to pull requests (PRs)
  7. Perl 6 documentation (docs)

Appendix

POD tools

  • perl6 –doc=MODULE # where ‘MODULE’ is ‘HTML’, ‘Text’, or other appropriate module
  • p6doc
  • perl6 –ll-exception

Major Perl 6 POD renderers

Day 12 – Avoiding User Namespace Pollution with Perl 6 Modules

A bit of trickery is needed

As Perl 5 module authors, we were able to allow selective exporting easily by use of the @EXPORT_OK array where we listed all the objects that the user of our module could import by name, thereby giving the user full control over possible namespace clashes. However, in spite of all the new goodies provided by Perl 6, that feature is not yet available to us Perl 6 module authors. (For more information see issue #127305 on https://rt.perl.org.)

But the good news is there is a way to get the same effect, albeit with a bit more effort. The trick is in using the full power of tags with the export trait available with every object in a module.

One might get a little glassy-eyed when reading the Perl 6 docs concerning module import and export, of which a portion of code is shown here:

unit module MyModule;
my package EXPORT::DEFAULT {
    our sub foo { ... }
}
my package EXPORT::other {
    our sub bar { ... }
}

where each object to be exported is enclosed in a named export category block. That method has serious limitations (in addition to requiring another set of child blocks):

  • the objects cannot appear in more than one grouping and
  • the objects cannot be assigned other tags via the is export adjective on individual objects.

Consequently, to emulate Perl 5’s @EXPORT_OK array, each object would have to be wrapped in a unique block and could not have multiple tags. But then the docs say that all that ugly package exporting code can be replaced by this much simpler code for each item to be exported:

sub foo is export { ... }
sub bar is export(:other) { ... }

And therein is our solution: We will use export tags to (1) allow the user to import only what he or she desires to import or (2) import everything. The module author can also be helpful by adding other tags which will allow importing sets of objects (but such sets are not always so easily determined in practice).

Note one restriction before we continue: only valid identifier names are allowed in the tags, so we can’t use sigils or other special characters as we could in Perl 5.

For an example we use a simple, but unusual, Perl 6 module with multiple objects to be exported (and all have the same name, probably not a good practice, but handy for demonstration of the disambiguation problem):

unit module Foo;
sub bar() is export { say "bar" }
constant $bar is export = 99;
constant %bar is export = (a => 1);
constant @bar is export = <a b>;

We can then have access to all of those objects by merely using the module in a program file:

use Foo;
bar;
say $bar;
say %bar;
say @bar;

Executing it yields:

bar
99
{a => 1}
[a b]

as expected. Now let’s modify the module by adding an export tag to the bar routine:

sub bar() is export(:bar) { say "bar" }

and then, when our program is executed, we get:

===SO6RRY!=== Error while compiling...
Undeclared routine:
    bar used at line 7. Did you mean 'VAR', 'bag'?

so we modify our use line in the program to:

use Foo :bar;

and execute the program again to get:

===SORRY!=== Error while compiling...
Variable '$bar' is not declared. Did you mean '&bar'?

It seems that as soon as we, the users, restrict the use statement with a tag, the non-tagged objects are not available! Now we, the authors, have two choices:

  • tag all objects with the same tag or
  • tag them with separate tags.

If we tag them the same, then all will be available—which is probably not a problem. However, that would defeat our goal of having unique tags.

If we name them with unique tags, we will need some way to distinguish them (remember, we can’t use their sigils), which leads us to a possible convention we show here in the final module:

unit module Foo;
sub bar() is export(:bar :SUB) { say "bar" }
constant $bar is export(:bar-s :CONST) = 99;
constant %bar is export(:bar-h :CONST) = (a => 1);
constant @bar is export(:bar-a :CONST) = <a b>;

Notice the suffixes on the non-subroutine objects (all constants) have their kind indicated by the ‘-X’ where the ‘X’ is ‘s’ for scalar, ‘h’ for hash, and ‘a’ for array. We have also added an additional tag which groups the objects into the general categories of subroutines or constants so the user could import, say, just the constants as a set, e.g, use Foo :CONST;.

Now we just change the use line to

use Foo :bar, :bar-s, :bar-h, :bar-a;

and get

bar
99
{a => 1}
[a b]

again.

Thus the good news is we can add a tag to the export trait that is the same as the name of the subroutine, but the sad news is we can’t use the appropriate sigil as in Perl 5 to disambiguate among objects with the same name. The solution to that is to use some convention as demonstrated above (akin to the ugly Hungarian notation in C) that will have the same effect.

Of course in this somewhat-contrived example, the user could have imported all the objects at once by using the special, automatically-defined tag `:ALL:

use Foo :ALL;

but the author has provided users of the module complete flexibility in its application for them.

Conclusion

We now have a way to protect the user’s namespace by requiring him or her to selectively import objects by “name” (or perhaps other subsets of objects) unless the user chooses to import everything. The only downsides I see are:

  • the extra effort required by the module author to explicitly tag all exportable objects and
  • the restriction on selecting an appropriate tag for different objects with the same name.

I love Perl 6, and the old Perl motto TIMTOWTDI still holds true as we’ve just seen!

Merry Christmas! // Happy Holidays! I hope you will enjoy the gift of Perl 6 throughout the coming year!


Note: Thanks to Moritz Lenz (IRC #perl6: user moritz) for constructive comments about the bad practice of exporting variables—hence the example now exports constants instead.