Day 2 – Like Perls in a Pod: document everything (and test the documentation)

Christmas season was approaching, and Santa was in a gloomy mood. His inbox was full with letters from boys and girls coming from all over.

But.

Were they letter to Santa? Was the kid properly identified by signature, so that you sent the gifts to the proper person and not someone else who might not deserve them? Were them addressed to Santa, and not any of those impostors, the Easter Bunny, or, even worse, the Three So-Called-I-don’t-know-why-Wise-Men from Orient? Worst of all, did he personally have to check all that stuff all by his royal and hallowed self?

No.

Perl 6 came to the rescue with the following grammar:

unit grammar Santa-Letter;

token TOP { <dear> \v+ <paragraph> [\v+ <paragraph>]* \v+ <signature>\v*}
token paragraph { <superword>[ \h+ <superword>]+ }
token superword { <word> | <enhanced-word> }
token word { \w+ }
token enhanced-word { <word> [\,|\.|\:] }
token dear {Dear \h+ [S|s]anta [\,|\:]? }
token signature {\h+ \w+ \h* \w* }

This unit declares a letter to Santa as a salutation, followed by one or more paragraphs, and finally, a signature, which should be preceded by an horizontal whitespace as indicated by \h.

Letters such as this one:

Dear Santa:

This year I have been a really good boy, I have been in all Squashathons.

So I want a plush Camelia studded with diamonds.

 JJ

A simple script will use that grammar and get the signature in a single letter:

use Santa-Letter;

sub MAIN ( Str $file = "letter.txt" ) {
    my $letter =$file.IO.slurp;
    my $parsed = Santa-Letter.parse($letter);
    say $parsed<signature>.trim;
}

That was good and well, but Santa needed to get that data into the North Pole’s CRM together with the letter and index everything up, and at the same time he had to deal with suppliers for whom the trade wars had brought havoc… So he he called his closest IT elf and asked him, in so many words, to do that kind of thing.

After that speech, the IT elf stood there, his ears aquiver.

“What?”, Santa growled. In a hallowed way, of course.

The pointed part of the ears reddened and, with the quivering, irradiated heat so that a small icicle melted and fell down to the earth.

“You can read the source, right?”

Rudolf, who had been awakened by the noise of the icicle melting, because that was one of his superpowers, intervened

Most people can read source, but everyone can read the documentation.

Said Rudolph.

“And everyone should write that documentation, too”, admonished, bobbing his head with the red-tipped nose on the front.

Santa mumbled, but eventually checked out the master branch of his Santa-Letter grammar and set down to work on it. Using, of course, Pod 6

Pod 6 stands for “Plain Old documentation for Perl 6”

And it is (clearly) not an acronym. Pod6 is a DSL that helps Perl 6 coders write documentation. It’s a markup language that uses = to start commands and for paragraph-level markup. We’ll get to that, but for the time being, Santa realized that one of the best things was how it integrates with Perl 6 itself. So he did a second iteration of his examining program thus:

#| This reads a letter file
sub MAIN ( Str $file = "letter.txt" ) {
    my $letter =$file.IO.slurp;
    my $parsed = Santa-Letter.parse($letter);
    say $parsed<signature>.trim;
}

There’s a funny sign, |, in that comment. That sign ties it to the code after the comment. And in this case, it’s the MAIN sub.

Santa released to production the program. The IT elf tried to run the program,

./get-signed.p6 --help

and he obtained:

Usage:
  ./get-signed.p6 [] -- This reads a letter file

“Some documentation is better than no documentation”, he thought. But that was not nearly enough. He entered the North Pole ticketing system, based entirely in free software, and he requested more documentation and assigned the task to Santa. Santa protested loudly, but complied.

#|{ This reads a letter file in text format.
With no arguments, it will read the C<letter.txt> file.
}
sub MAIN ( Str $file = "letter.txt" ) {
    my $letter =$file.IO.slurp;
    my $parsed = Santa-Letter.parse($letter);
    say $parsed<signature>.trim;
    say $=pod[0].perl;
}

This printed the same message when invoked with --help. And it was documentation. When running

perl6 --doc get-signed.p6

it printed:

sub MAIN(
	Str $file = "letter.txt", 
)
This reads a letter file in text format. With no arguments, it will read the C file.

So Perl 6 understands the comment and the code attached to it, and automatically pretty-prints both. Documenting a routine is as easy as this.

Besides, when run on an actual file, the last sentence kicked it, and it printed:

Pod::Block::Declarator.new(WHEREFORE => sub MAIN (Str $file = "letter.txt") { #`(Sub|81308800) ... }, config => {}, contents => [])

Unlike other DSLs used for comments in other languages, such as Markdown or Pod itself in Perl 5, Pod 6 not only is a DSL for comments, it’s part of Perl 6 itself, and thus, it’s interpreted by the Perl 6 parser, its internal structures available for introspection
in the $=pod variable. In this case, the comment is aPod::Block::Declarator, and that data structure includes the WHEREFORE key which contains the declared function and the comment. However, contents and config are empty. Which they shouldn’t.

What’s more, the little bit of actual formatting used in the comment does not work. Not to mention the actual module was not really documented. Now it was Santa who was not happy.

Adding documentation to a module.

Writing documentation is probably the first thing you should do before writing the actual code. Documentation is for the module clients, but first and foremost, it’s a guide for the author, a roadmap of what the module should do and how it should do it. As seen above, documenting individual methods or routines is quite easy with Pod 6; however, a big picture view of the module is also convenient. And here’s the Pod for Santa-Letter

=begin pod

=head1 NAME

Santa-Letter - A grammar for letters to Santa for the L<Perl 6 Advent Calendar|https://perl6advent.wordpress.com>

=head1 SYNOPSIS

Parses letters formatted nicely and written by all good kids in the world.

=end pod

Conveniently placed at the end of the file, when invoked with perl6 --doc Santa-Letter.pm6, or simply perl6 --doc Santa-Letter if it has been installed, or even p6doc Santa-Letter if the perl6/docis present, will write something like:

NAME

Santa-Letter - A grammar for letters to Santa for the Perl 6 Advent
Calendar

SYNOPSIS

Parses letters formatted nicely and written by all good kids in the
world.

But you will notice here that there was a piece of markup that has been eliminated in this type of output. L creates links, but it obviously does so only if the output format supports that. So let’s try one of those:

perl6 --doc=HTML Santa-Letter.pm6

will output a good amount of code, among which this line:

Santa-Letter - A grammar for letters to Santa for the <a href="https://perl6advent.wordpress.com">Perl 6 Advent Calendar</a>

clearly shows the output of the line that includes the link.

As a matter of fact, the above command will use the Pod::To::HTML module to convert the Pod data structures to HTML. Using any other value after the = will call the corresponding Pod::To::X module, and there are many of them available on the ecosystem. For instance,Pod::To::Pager will use the system’s pager to make stuff a bit more beautiful.

perl6 --doc=Pager Santa-Letter.pm6 

will result in this

pager

This documentation, besides, follows the convention used in all modules. NAME should describe the name and a short oneliner that tells what the module is about, while SYNOPSIS includes a longer description. While that’s good, a real piece of documentation should include examples.

=begin code

use Santa-Letter;

say Santa-Letter.parse("Dear Santa\nAll I want for Christmas\nIs you\n Mariah");

=end code

Examples are included in code blocks, which from the point of view of Pod6, are Pod::Block::Codeobjects. Which is a nice thing, actually. Let’s add this little snippet of code to our grammar:

our $pod = $=pod[0];

Grammars are classes, and they have class-scoped variables. We can’t export the $=pod variable to avoid clashing with others, but we can assign it to a class (grammar) scoped variable (thus the our) it and then use it from our program this way:

say $Santa-Letter::pod.perl;

Or, even better, install Data::Dump and write something like this:

say Dump( $Santa-Letter::pod, :indent(4), :3max-recursion );

which uses the pod class variable we have declared, and prints it this way:

structure

This tree, which could be called the POM (Pod Object Model), includes, besides the known name and config metadata that goes with every block, an array of Pod6 blocks at the same level. Every one has the generic attributes plus specific attributes, like level in the case of headings. Anyway, the interesting thing is that the code itself we are using as an example is available as contents of the Pod::Block::Code object.

“Hum”, thought Santa. We could do one better with this. Can we actually check that the included code works? Yes we can! Let’s expand the SYNOPSIS section:

=head1 SYNOPSIS

Parses letters formatted nicely and written by all good kids in the world.

=begin code

use Santa-Letter;

say Santa-Letter.parse("Dear Santa\nAll I want for Christmas\nIs you\n Mariah");

=end code

You can also access particular elements in the letter, as long as they are included on the grammar

    my $letter="Dear Santa,\nI have not been that good.\nJust a paper clip will do\n Donald"
    say Santa-Letter.parse($letter)<signature>

Also

=for code :notest :reason("Variable defined above")
say "The letter signed by ", Santa-Letter.parse($letter),
    " has ", Santa-Letter.parse($letter).elems, " paragraphs";
    
=end pod

Code can be represented in different ways in a Pod. The first is known; the second uses simply indentation, à la Markdown, to denote the same thing. We can also use =for paragraph blocks, which is declared in this case with the code type and will continue until the next blank line. It’s an abbreviated way that does not need the =end directive. But there’s something more there: the configuration variables :notest :reason("Variable defined above"). These configuration variables are arbitrary, and we can add as many as we want. They will go to the config attribute of the block, and we can work with them. That’s exactly what we will do in this script that will process the code examples:

for $Santa-Letter::pod.contents -> $block {
    next if $block !~~ Pod::Block::Code;
    if $block.config<notest> {
        say "→ Block\n\t"~ $block.contents
            ~ "\n\t❈ Not tested since \'" ~ $block.config<reason> ~ "\'";
    } else {
        my $code = $block.contents.join("");
        say "→ Block\n\t"~ $block.contents;
        try {
            EVAL $code;
        }
        if ( $! ) {
            say "\n\t✘ Produces error \"$!\"", "\n" xx 2;
        } else {
            say "✔ is OK\n";
        }
    }
}

As we have seen in the structure above, the contents attribute will include an array of first-level Pod blocks, which in our case include all the three blocks we want to evaluate (or maybe not). Non-code blocks are skipped (but could be checked for spelling, too). We do two interesting things here: we check for the notest flag in the configuration via $block.config, and we print some note if that’s the case, but if it should be tested, then it’s EVALed (we need the use MONKEY-SEE-NO-EVAL pragma for that).

Santa runs that on the documentation, and lo and behold!

→ Block
	my $letter="Dear Santa,\nI have not been that good.\nJust a paper clip will do\n Donald"
say Santa-Letter.parse($letter)

	✘ Produces error "Two terms in a row across lines (missing semicolon or comma?)"(
 
)

He was at once happy and humbled. A simple semicolon was spoiling the quality of the examples. It’s always the semicolon. He put the semicolon back in the examples, and the module documentation passed the test with flying colors.

Back to production

Provided with this documented module, the IT elf was moderately happy and his ears stopped quivering and reddening. He could also use documentation for every one of the tokens, but enough was enough and at least he had some examples to get the application going. Now he would have to write the bridge between the letter-receiving microservice and the customer relationship macroservice. He would probably use Cro for that, but that’s a topic for another day.
Meanwhile, Rudolf was soundly asleep, and Santa was back to his Santaish occupations. All was good and well in the North Pole.

Day 3 – LetterOps with Perl6

Scale

“Scale! Scale is everything!”.

Elves scattered in all directions when the booming voice of Santa reached them.

“This operation is prepared for, what, thirty four children? And now we have zillions of them! And adults are sending letters too!”

Buzzius the elf stepped forward and spurted “But now we have computers!”, darting back again to his elvish pursuits.

“What good are they? Pray tell me, what can I do if I still have to read every single letter?”.

Diodius the elf briefly raised his head from his hiding place and said “Tell the children to send a letter in a text file”.

Santa stopped yelling and scratched his well-bearded chin. “I can do that”. Early children adopters sent a letter just like this one.

Dear Santa: I have been a good boy so I want you to bring me a collection of scythes and an ocean liner with a captain and a purser and a time travel machine and instructions to operate it and I know I haven't been so good at times but that is why I'm asking the time machine so that I can make it good and well and also find out what happened on July 13th which I completely forgot.

“I can do that?”. Santa repeated to himself. He would have to extract a list of gifts out of that single-line mess. For instance, dividing it by and.

And, of course, using Perl 6, which being able to use as a variable, and even runic our $ᚣ = True was his favorite language. In a single line you can get all the chunks obtaining something like this:

[ "Dear Santa: I have been a good boy so I want you to bring me a collection of scythes", "an ocean liner with a captain", "a purser", "a time travel machine", "instructions to operate it", "I know I haven't been so good at times but that is why I'm asking the time machine so that I can make it good", "well", "also find out what happened on July 13th which I completely forgot.\n" ]

The /\s* «and» \s*/ regexp took the ands and also trimmed spaces, creating a set of sentences. And these sentences might or might not contain something the customer wanted Santa to bring. Which made Santa start roaring again. “Scale and structure! We need to scale and we need structure!”

Markdown to the rescue

Marcius pitched in. “Everybody knows Markdown. It’s text, with a few marks thrown in for structure.”

Oakius, who was working towards his promotion to Elf, Second class, said. “Use the elvish-est language, Elm. You know, it’s elf but for a letter”

“I can do that”, said Santa. Elves loved his can do approach. So he installed the whole thing and did this little program

Santa was quiet for about 30 seconds. And then his roaring could be heard again.

“Never, you hear me? Never I want to hear again about this spawn from the Easter Bunny or other evil creatures”.

Those elves nearest the screen observed lots of red, but not nice red, and nothing resembling working code. So they gave Rudolph the (nice) Red Nose Reindeer a note, which he dutifully carried pricked in one of his smaller antlers.

“Should we go back to Perl6 then?”

Processing Markdown with Perl6

Santa Found Text::Markdown, which he promptly installed with

zef install Text::Markdown

It had Text, it had Markdown, it promised to process it, that was all he needed. So he got word to his customer base that markdown was going to be needed this year if you wanted this guy to go down your chimney with a burlap bag with nice stuff in it.

Once again, early adopters answered with this

# Dear Santa

I have been a good boy so I want you to bring me a collection of
scythes and an ocean liner with a captain and a purser and a time
travel machine and instructions to operate it and I know I haven't
been so good at times but that is why I'm asking the time machine so
that I can make it good and well and also find out what happened on
July 13th which I completely forgot.

Well, it is Markdown, is is not? It’s properly addressed and all. “Properly addressing a letter is important”, Santa said aloud, in a not-quite-yell that only startled Rudolph, which was the only one hanging around. “It gives structure. Let us check whether letters have this”.

“Wow!” Said Santa. And then, “Wow”. Just a few lines of code, one to read and understand the structure of the document, another one to check if there is at least one that is a heading. It will say True if that is the case. And it was true.

Santa was happy for a tiny while. He scratched the scruff of the neck of Rudolph, who was kind of surprised by this. Then he stopped doing it. Rudolph looked up and backed his hind legs just this tiny bit, feeling unhappiness.

More structure is needed.

Santa had found this letter:

# Dude

## Welll...

I have been a naughty person


## Requests

Well...

Proper addressing and everything, he could not waste his time with persons that had not been good. Scale. And resources. Resources should be spent only in good persons, not in bad persons. Bad persons are bad, and that’s that. So went back to coding, Rudolph slipped away looking for lichen candy or whatever, and he produced this:

Santa was kind of proud of the trick that extracted the paragraphs after the second heading, as well as the fact that he had been able to put to good use the Thorn letter, which he loved. He also loved functional programming, having cut his teeth in Lisp. So he created this flip-switch that is initially false but flips on when the element it is dealing with is a heading and its level is two. He was also happy that he could do this kind of thing with the structured layered on top of the text by the marks.

Besides, he could check whether the word “good” was present in any of the paragraphs between that heading (Behavior) and the next. And any is so cool. It is enough that one of the paragraph mentions good. The last line will first return an array of Boolean values, and will eventually say True if just one of them includes good. False otherwise. Good for culling the good from the bad.

Santa was happ-y. -Ier. But still.

The toys are the important thing here.

So what he actually wanted was a list of the toys. After requesting, once again, a change of letter format, which he could do because he was Santa and everyone wanted his free stuff for Christmas, he started to receive letters with this structure:

# Dear Santa

## Behavior

I have been a good boy 


## Requests

And this is what I want

 - scythes 
 - an ocean liner with a captain and a purser
 - a time travel machine and instructions to operate it 

What they lack in spontaneity they have in structure. And structure is good. You can get a list of requests thus:

That is really an unsaintly list of chained list processing expressions. And this sentence before this one has an list of list mentions that is almost as bad. But let us see what is going on there.

First thing in the list, we take only what comes after the Requests heading, using regular expressions and stuff. We could have probably pared it down to a transformation to Str but we would have lost the structure. And structure is important, Santa is never tired of repeating that. Next we extract only those elements that are actually a list, taking out all the fluff.

And it so happens that there is such a thing as too much structure. The list has elements that have elements that have elements in it.

That, or Text::Markdown could do with a big makeover. Which is what the author of this post is putting on his particular wish list.

Not there yet

But almost. We have the list, and now Santa finds things like time travel machines and Mondays and things like that. He cannot order Mondays in the elf factory. He would have to read every single list of things. But no worries. That can be taken care of, too:

Simply enough, this program goes over the saved list of items in the wish list, and checks for product-ness. Is it a product? It goes. Are you asking for last Friday evening, which you completely missed? It does not, and don’t you dare to waste Santa’s time, boy.

The gist of the thing is in the Wikidata query, which uses the brand-new Wikidata::API module. This module just sends stuff to the Wikidata API and returns it as an object. Believe it or not, that is what the SPARQL query does: inserts the item name into the query, makes the query, and returns true if the number of returned elements is not zero. Productness at your fingertips! In a few lines of code! Now he could just chain all the stuff together and obtain from a letter containing this

 - Morning sickness
 - Scythe
 - Mug

Just the two of them which you can actually order from your local, downtown, mom and pop shop, which is where Santa actually goes to secretly buy all the stuff because he buys in bulk and he gets a pretty good deal.

Santa smiled, and a loud cheer erupted from the crowd of elves, reindeers, and a couple of puffins that were there for no good reason. They then set down to

Wrap up

Santa and Perl 6 are a good match, simply because they both came in Christmas time. Santa finds you can do lots of useful things with it, by itself or by using one of the fine modules that have become available lately.

The author of this, however, will include in his letter to Santa some help to carry ahead with the two modules used in this post, maintained by him, and which need more experienced coders to test, extend and maybe rewrite from scratch. But he is happy to see that mundane and slightly divine things like processing letters to Santa can be done straight away using Perl6. And you should it too.

Code and samples for this post are available from GitHub. Also this text. Help and suggestion are very much welcome.