Day 8 – Adventures in NQP Land: Hacking the Rakudo Compiler

With apologies to the old Christmas classic, “The Twelve Days of Christmas,” I give you the first line of a Perl 6 version:

On the first day of Christmas, my true love gave to me, a Perl table in a pod tree…

But the table I got wasn’t very pretty!

Background

My first real contact with Perl 6 came in the spring of 2015 when I decided to check on its status and found it was ready for prime time. After getting some experience with the language I started contributing to the docs in places where I could help. One of my first contributions to the docs was to clean up one of the tables which was not rendering nicely . During my experiments with pod tables on my local host I tried the following table:

      

        

        
=begin table
      
      

        

        
-r0c0  r0c1
      
      

        

        
=end table
      

which caused Perl 6 to throw an ugly, LTA (less than awesome) exception message:

      

        

        
"===SORRY!=== Cannot iterate object with P6opaque representation"
      

I worked around the problem but it nagged at me so I started investigating the guts of pod and tables. That led me to the source of the problem in github.com/rakudo/src/Perl6/Pod.nqp.

In fact, the real problem for many pod table issues turned out to be in that file.

Not Quite Perl (NQP)

nqp is an intermediate language used to build the Rakudo Perl 6 compiler. Its repository is found here. The rest of this article is about modifying nqp code in the rakudo compiler found in its repository here. Rakudo also has a website here.

Before getting too far I first read the available information about Rakudo and NQP here:

• Slides from Jonathan Worthington’s (JWs) course on Rakudo and NQP Internals
• NQP opcodes
• NQP built-in routines

Then I started practicing nqp coding by writing and running small nqp files like this (file “hello.nqp”):

      

        

        
say("Hello, world!");
      

which, when executed, gave the expected results:

      

        

        
$ nqp hello.nqp
      
      

        

        
Hello, world!
      

Note that say() is one of the few nqp opcodes that doesn’t require the nqp:: prefix.

Into the trenches

The purpose of the Perl6::Pod class, contained in the rakudo/src/Perl6/Pod.nqp file, is to take pod grammar matches and transform them into Perl 6 pod class definitions, in rakudo/src/core/Pod.pm, for further handling by renderers in Perl 6 land. For tables that means anything represented in any legal pod form as described by the Perl 6 documentation design Synopsis S26, the Perl 6 test suite specs, and the Perl 6 docs has to be transformed into a Perl 6 Pod::Block::Table class object with this form as described in file rakudo/src/core/Pod.pm:

      

        

        
configuration information
      
      

        

        
a header line with N cells
      
      

        

        
M content lines, each with N cells
      

I wanted the nqp table pod handling to be robust and able to automatically fix some format issues (with a warning to the author) or throw an exception (gracefully) with detailed information of problems to enable the author to fix the pod input.

Workspace and tools

I needed two cloned repositories: rakudo and roast. I also needed forks of those same repositories on github so I could create pull requests (PRs) for my changes. I found a very handy Perl 5 tool in CPAN module App::GitGot. Using got allowed me to easily set up all four repos. (Note that got requires that its target repo not exist either in the desired local directory or the user’s github account.) After configuring got I went to a suitable directory to contain both repos and executed the following:

      

        

        
got fork https://github.com/rakudo/rakudo.git
      
      

        

        
got fork https://github.com/perl6/roast.git
      

which resulted in a subdirectories rakudo and roast containing the cloned repos and new forks of rakudo and roast on my github account. In the rakudo directory one can see the default setup for easy creation of PRs:

      

        

        
$ git remote -v
      
      

        

        
origin  git@github.com:tbrowder/rakudo.git (fetch)
      
      

        

        
origin  git@github.com:tbrowder/rakudo.git (push)
      
      

        

        
upstream    https://github.com/rakudo/rakudo.git (fetch)
      
      

        

        
upstream    https://github.com/rakudo/rakudo.git (push)
      

There are similar results in the roast repo.

Next, I renamed the roast repo as a subdirectory of rakudo (“rakudo/t/spec”) so it functions as a subgit of the local rakudo.

Finally, I created several bash scripts to ease configuring rakudo for installation in the local repo directory, setting the environment, and running tests:

• rakudo-local-config.sh
• run-table-tests.sh
• set-rakudo-envvars.sh

(See all scripts mentioned here at https://github.com/tbrowder/nqp-tools.)

To complete the local working environment you will need to install some local modules so you must change your path and install a local copy of the zef installer. Follow these steps in your rakudo directory (from advice from @Zoffix):

      

        

        
git clone https://github.com/ugexe/zef
      
      

        

        
export PATH=`pwd`/install/bin:$PATH
      
      

        

        
cd zef; perl6 -Ilib bin/zef install .
      
      

        

        
cd ..
      
      

        

        
export PATH=`pwd`/install/share/perl6/site/bin:$PATH
      
      

        

        
zef install Inline::Perl5
      

Then install any other module you need, e.g.,

      

        

        
zef install Debugger::UI::CommandLine
      
      

        

        
zef install Grammar::Debugger
      

Hacking

Now start hacking away. When ready for a build, execute:

      

        

        
make
      
      

        

        
make install
      

The make install step is critical because otherwise, with the local environment we set up, the new Perl 6 executables won’t be found.

Debugging for me was laborious, with rebuilds taking about three minutes each. The debugger (perl6-debug-m) would have been very useful but I could not install the required Debbugger::UI::CommandLine module so it would be recognized by the locally-installed perl6-debug-m. The primary method I used was inserted print statements plus using the --ll-exception option of perl6. Of major note, though, is that this author is a Perl 6 newbie and made lots of mistakes, and did not always remember the fixes, hence this article. (Note I likely would have used the debugging tools but, at the time I started, I did not ask for help and did not have the advice provided shown above.)

Testing

It goes without saying that a good PR will include tests for the changes. I always create a roast branch with the same name as my rakudo branch. Then I submit both PRs and I refer to the roast PR in the rakudo PR and vice versa. I note for the roast PR that it requires the companion rakudo PR for it to pass all tests.

See Ref. 5 for much more detail on specialized test scripts for fudging and other esoteric testing matters.

Documentation

I try to keep the Perl 6 pod table documentation current with my fixes.

NQP lessons learned

• LTA error messages are a fact of life, e.g., “Cannot invoke this object…”, which can be caused by many things, including a misspelled identifier (NQP issue filed, early report is it may be impossible to fix anytime soon).

• Ensure all nqp opcodes have the nqp:: prefix (except the few built-ins)

• Practice with new code in an nqp-only sand-box.

Success!

I have now had two major Perl 6 POD (and roast) PRs accepted and merged, and am working on one more “easy” one which I should finish this week. The PRs are:

1. Rakudo PR #1240

The Rakudo PR provided fixes for RTs #124403, #128221, #132341, #13248, and #129862. It was accompanied by roast PR #353.

The PR allowed the problem table above to be rendered properly. It also added warnings for questionable tables, added Rakudo environment variables RAKUDO_POD6_TABLE_DEBUG to aid users in debugging tables (see docs, User Debugging), and allows short rows with empty columns to be rendered properly.

1. Rakudo PR #1287

The Rakudo PR provided a fix for Rakudo repo issue #1282. It was accompanied by roast PR #361. (Note that roast PR #361 is not yet merged.)

The PR allows table visual column separators (‘|’) and (‘+’) to be used as cell data by escaping them in the pod source.

Summary

• Perl 6 pod is a great improvement over Perl 5, but it is still not fully implemented.

• Working in the bowels of Rakudo Perl is rewarding (and fun), but prepare to get your hands dirty!

• The Perl 6 community is a great group to be associated with.

• I love Rakudo Perl 6.

Merry Christmas and Happy Hacking!

Credits

Any successful inputs I have made are due to all the Perl 6 core developers and the friendly folks on IRC (#perl6, #perl6-dev).

References

1. JWs Perl 6 debugger Advent article
2. JWs Rakudo debugger module Debugger::UI::CommandLine
3. JWs grammar debugger module Grammar::Debugger
4. Testing Rakudo
5. Contributing to roast
6. Github guide to pull requests (PRs)
7. Perl 6 documentation (docs)

Appendix

POD tools

• perl6 –doc=MODULE # where ‘MODULE’ is ‘HTML’, ‘Text’, or other appropriate module
• p6doc
• perl6 –ll-exception

Major Perl 6 POD renderers

• Pod::To::Text (part of the rakudo core)
• Pod::To::HTML