Day 24 – Make It Snow

Hello again, fellow sixers! Today I’d like to take the opportunity to highlight a little module of mine that has grown up in some significant ways this year. It’s called Terminal::Print and I’m suspecting you might already have a hint of what it can do just from the name. I’ve learned a lot from writing this module and I hope to share a few of my takeaways.

Concurrency is hard

Earlier in the year I decided to finally try to tackle multi-threading in Terminal::Print and… succeeded more or less, but rather miserably. I wrapped the access to the underlying grid (a two-dimensional array of Cell objects) in a react block and had change-cell and print-cell emit their respective actions on a Supply. The react block then handled these actions. Rather slowly, unfortunately.

Yet, there was hope. After jnthn++ fixed a constructor bug in OO::Monitors I was able to remove all the crufty hand-crafted handling code and instead ensure that method calls to the Terminal::Print::Grid object would only run in a single thread at any given time. (This is the class which holds the two-dimensional array mentioned before and was likewise the site of my react block experiment).

Here below are the necessary changes:

- unit class Terminal::Print::Grid;
+ use OO::Monitors;
+ unit monitor Terminal::Print::Grid;

This shift not only improved the readability and robustness of the code, it was significantly faster! Win! To me this is really an amazing dynamic of Perl 6. jnthn’s brilliant, twisted mind can write a meta-programming module that makes it dead simple for me to add concurrency guarantees at a specific layer of my library. My library in turn makes it dead simple to print from multiple threads at once on the screen! It’s whipuptitude enhancers all the the way down!

That said, our example today will not be writing from multiple threads. For some example code that utilizes async, I point you to examples/async.p6 and examples/matrix-ish.p6.

Widget Hero

Terminal::Print is really my first open source library in the sense that it is the first time that I have started my own module from scratch with the specific goal of filling a gap in a given language’s ecosystem. It is also the first time that I am not the sole contributor! I would be remiss not to mention japhb++ in this post, who has contributed a great deal in a relatively short amount of time.

In addition to all the performance related work and the introduction of a span-oriented printing mechanism, japhb’s work on widgets especially deserves its own post! For now let’s just say that it has been a real pleasure to see the codebase grow and extend even as I have been too distracted to do much good. My takeaway here is a new personal milestone in my participation in libre/open software (my first core contributor!) that reinforces all the positive dynamics it can have on a code base.

Oh, and I’ll just leave this here as a teaser of what the widget work has in store for us:

rpg-ui-p6

You can check it out in real-time and read the code at examples/rpg-ui.p6.

Snow on the Golf Course

Now you are probably wondering, where is the darn, snow! Well, here we go! The full code with syntax highlighting is available in examples/snowfall.p6. I will be going through it step by step below.

use Terminal::Print;

class Coord {
    has Int $.x is rw where * <= T.columns = 0;
    has Int $.y is rw where * <= T.rows = 0 ;
}

Here we import Terminal::Print. The library takes the position that when you import it somewhere, you are planning to print to the screen. To this end we export an instantiated Terminal::Print object into the importer’s lexical scope as T. This allows me to immediately start clarifying the x and y boundaries of our coordinate system based on run-time values derived from the current terminal window.

class Snowflake {
    has $.flake = ('❆','❅','❄').roll;
    has $.pos = Coord.new;
}

sub create-flake {
    state @cols = ^T.columns .pick(*); # shuffled
    if +@cols > 0 {
        my $rand-x = @cols.pop;
        my $start-pos = Coord.new: x => $rand-x;
        return Snowflake.new: pos => $start-pos;
    } else {
        @cols = ^T.columns .pick(*);
        return create-flake;
    }
}

Here we create an extremely simple Snowflake class. What is nice here is that we can leverage the default value of the $.flake attribute to always be random at construction time.

Then in create-flake we are composing a way to make sure we have hit every x coordinate as a starting point for the snowfall. Whenever create-flake gets called, we pop a random x coordinate out of the @cols state variable. The state variable enables this cool approach because we can manually fill @cols with a new randomized set of our available x coordinates once it is depleted.

draw( -> $promise {

start {
    my @flakes = create-flake() xx T.columns;
    my @falling;
    
    Promise.at(now + 33).then: { $promise.keep };
    loop {
        # how fast is the snowfall?
        sleep 0.1; 
    
        if (+@flakes) {
            # how heavy is the snowfall?
            my $limit = @flakes > 2 ?? 2            
                                    !! +@flakes;
            # can include 0, but then *cannot* exclude $limit!
            @falling.push: |(@flakes.pop xx (0..$limit).roll);  
        } else {
            @flakes = create-flake() xx T.columns;
        }
    
        for @falling.kv -> $idx, $flake {
            with $flake.pos.y -> $y {
                if $y > 0 {
                    T.print-cell: $flake.pos.x, ($flake.pos.y - 1), ' ';
                }

                if $y < T.rows {
                    T.print-cell: $flake.pos.x, $flake.pos.y, $flake.flake;            
                }

                try {
                    $flake.pos.y += 1;
                    CATCH {
                        # the flake has fallen all the way
                        # remove it and carry on!
                        @falling.splice($idx,1,());
                        .resume;
                    }
                }
            }
        }
    }
}

});

Let’s unpack that a bit, shall we?

So the first thing to explain is draw. This is a handy helper routine that is also imported into the current lexical scope. It takes as its single argument a block which accepts a Promise. The block should include a start block so that keeping the argument promise works as expected. The implementation of draw is shockingly simple.

So draw is really just short-hand for making sure the screen is set up and torn down properly. It leverages promises as (I’m told) a “conv-var” which according to the Promises spec might be an abuse of promises. I’m not very futzed about it, to be honest, since it suits my needs quite well.

This approach also makes it quite easy to create a “time limit” for the snowfall by scheduling a promise to be kept at now + 33 — thirty three seconds from when the loop starts. then we keep the promise and draw shuts down the screen for us. This makes “escape” logic for your screensavers quite easy to implement (note that SIGINT also restores your screen properly. The most basic exit strategy works as expected, too :) ).

The rest is pretty straightforward, though I’d point to the try block as a slightly clever (but not dangerously so) combination of where constraints on Coord‘s attributes and Perl 6’s resumable exceptions.

Make it snow!

And so, sixers, I bid you farewell for today with a little unconditional love from ab5tract’s curious little corner of the universe. Cheers!

snowfall-p6

Day 11 – The Source will be with you, always

Reportings from a Learnathon

This past weekend I had the pleasure of hosting a Perl 6 learnathon with a friend who contacted me specifically to have a night of absorbing this new version of Perl. I thought it might be interesting to share some of what we learned during the process. I will begin with by explaining the single line of code which ended up and then show you some examples of where our evening took us.

Pick Whatever, rotor by five

As we opened our laptops, I was excited to show the newest example I had pushed to my Terminal::Print project. It’s taken quite some time to achieve, but asynchronous printing is now working with this module. It’s not fast, yet, but my long saught multi-threaded “Matrix-ish” example is working. Each column is being printed from an individual thread. This approach of spinning off a bunch of new threads and then throwing them away is not efficient, but as a proof of concept I find it very exciting.

pick-rotor

This line contains a few things that inspired questions. The first is the precise meaning of pick(*), which here means that we want to produce a randomized version of @columns. You can think of the Whatever here as a meaning “as many as possible”. It triggers a different multi method code path which knows to use the invoking objects’s own array size as the length of the list to pick.

The second part to explain was rotor. This is another one of those Perl 6 English-isms that at first felt quite strange but quickly became a favorite as I began visualizing a huge turbine rotor-ing all of my lists into whatever shape I desire whenever I use it. In this case, I want to assemble a list of 5-element arrays from a random version of @columns. By default, rotor will only give you back fully formed 5-element arrays, but by passing the :partial we trigger the multi-method form that will include a non 5-element array if the length of @columns is not divisible by 5 (‘Divisibility’ is easily queried Perl 6, by the way. We phrase it as $n %% 5.)

Put another way: rotor is a list-of-lists generator, taking one list and chopping it up into chunks according to your specifications. My friend mentioned that this resembles a question that he asks in interviews, inviting an investigation into the underlying implementation.

I’ve always considered Rakudo’s Perl 6-implemented-in-Perl 6 approach as a secret weapon that often goes overlooked in assessment of Perl 6’s viability. Even with the current reliance on NQP, there is a great deal of Perl 6 which is implemented in the language itself. To this end, I could immediately open src/core/Any.pm in an editor and continue explaining the language by reading the implementation of the language. Not just explaining the implementation, which can be accomplished by the right combination of language/implementation language and polyglot coverage. I mean explaining the language by looking at how it is used to implement itself, by following different threads from questions that arise as a result of looking at that code.

A word of caution

Now, I don’t mean to imply that one’s initial exposure to core can’t be a shocking experience. You are looking at Perl 6 code that is written according to constraints that don’t exist in perl6 which arise from not being fully bootstrapped and performance concerns. These are expressed in core by NQP and relative placement in the compilation process, on the one hand, and in prototyping and hoop jumping, on the other.

In other words: you are not expected to write code like this and core code does not represent an accurate representation of what Perl 6 code looks like in practice. It is 100% really Perl 6 code, though,  and if you look at NQP stuff as funky library calls, everything you are seeing is possible in your own code. You just don’t normally need to.

Down the rabbit hole

From src/core/Any.pm, here is the code for rotor:

any-rotor.png

And the code for pick:

pick-any

These are not the actual implementations, mind you. Those live in List.pm. But already these signatures inspire some questions

What is this |c thing in the signature?

This represents a Capture, which is an object representing the arguments passed into the method call. In this case it is being used such that no unpacking/containerization occurs. Rather we simply accept what we have been given and pass them as a single parameter to self.list.rotor without looking at them.

In the proto signature for pick we see that there is no name for the Capture, but rather a bare ‘|‘ which tells the method dispatcher that there can be zero or more arguments. Put another way: there are no mandatory arguments that apply to all the pick candidates.

What is this proto thing?

The answer is that it is a clarification that you can apply to your multi methods that constrains them to a specific “shape”.  It is commonly used in conjunction with a Capture, and in fact we see this in our pick example.

As the docs put it, a proto is ‘a way to formally declare commonalities between multi candidates’. The prototype for pick specifies that there are no mandatory arguments, but there might be. This is basically the default behavior for multi method dispatch, but here it allows us to specify the is nodal trait, which provides further information to the dispatcher relating to the list-y. Also due to bootstrapping issues, all multi methods built into Rakudo need an explicit proto.In practice we do not need either of these for our code. But they are there when you need them. One example of a trait that you might use regularly is the pure trait: when a given input has a guaranteed output, you can specify that it is pure and the dispatcher can return results for cached arguments without repeated calculations.

Midnight Golf

As promised, here are a few code examples from the learnathon.

stateful.pngstateful-results.png

This is using anonymous state variables, which are available to subroutines and methods but not to other Callables like blocks.  My friend shared my marvel at the power and flexibility of the Perl 6 parser to be able to handle statements like $--*5, when every character besides the 5 in that statement has a different meaning according to context. Meanwhile Perl 6 gives you defaults for your subroutine parameters by using the assignment operator.

Note that each bare $ in a subroutine creates/addresses a new, individual state variable. Some people will hate this, as they hate much that I appreciate about the language. These anonymous state variables are for situations where a name doesn’t matter, such as golfing on the command line. They can be confusing to get a full grasp of, though.

Here is another example we generated while exploring (and failing many times) to grasp the nuances of these dynamics.

more-state.pngmore-state-results.png

Gone is the anonymous state variable. This is by necessity, because you can only refer to an anonymous state variable once in a subroutine. We’ve switched $a to be optional. The parens around the state declaration are necessary because trying to declare a variable in terms of itself is undefined behavior and Perl 6 knows that.

The same thing, expressed in slight variations but with the same meaning:

more-more-state.png

The bottom example shows that the assignment to zero in our initial more-state is actually unnecessary. The middle shows creating a variable and binding it to a state var, which is a rougher way to get the same behavior as openly declaring a state variable. The top example shows what might be considered the ‘politely verbose’ option.

Concluding thoughts

I was hoping to share more from the evening with you, but it’s been a lot of words already and we’ve only scratched the surface of the first example we examined! Instead, I recommend that you spend some time with src/core in an editor,  the perl6 repl in a terminal, and #perl6 in an IRC client (or web browser) and just explore.

This language is deep. Opening src/core is like diving into the deep end of the ocean from the top of a Star Destroyer. The reward for your exploration of the language, however, is an increasing mastery of a programming language that is designed according to a human-centric philosophy and which approaches a level of expressivity that, some would argue, is unparalleled in the world of computing.

 

Day 4 – Composers, Coercers, and the Case of a Camel’s Curious Corner

“Every programming language has its curious corners, it’s just that some languages’ corners are curiouser than others’.”

When viewed from space, flame wars over programming languages are all about flinging various code or other examples around in order to prompt Yet Another Potentially Awkward Explanation (YAPAE) from the other side.

Even God has to give a YAPAE every once in a while

Even God has to give a YAPAE every once in a while… [image courtesy of xkcd++]

Everyone seems to know that Perl has corners. What is perhaps less famous than these corners themselves is the fact that their YAPAEs by and large make total sense. It may be a very Perlish form of sense, quite particular to Perl 5’s associativity, context, or precedence rules, but the reasoning always works out right. The language itself has a pretty simple set of rules, it’s the interactions of those rules that any good YAPAE code snippet will actually be highlighting.

Yet if the explanation has the potential to make sense, then it is only ever potentially awkward. Behind every YAPAE is the story of a decision in language design, a place where some of the implicit rules of operation suddenly become explicit1. As such they form the intersecting seams of design choices that were made at the core language level. The explanations are never all that awkward to people who understand those choices and agree with them. I like to think of these seams in the language as “curious corners,” which refers both to the surface-level surprise that comes from a YAPAE as well as my own curiosity in the ways different languages handle their seams that show.

Perl 5 does some pretty incredible things with what syntactically amount to a few rules about precedence, associativity, and context. There’s a lot more to the language, to be sure, but when something blows up in your face it is almost one of these categories that you’ve botched. As such, almost any of syntactical surprise in Perl 5 can be solved with liberal application of parentheses. (If Perl 5 is the duct tape of the internet, then the parenthesis is the duct tape of Perl 5). These usually work together so well that you only notice them where they collide in strange ways, and once you get bitten you start to know better. That said, it was these curious corners which led to the development of a new Perl, which would have different rules and different interactions of those rules. Some of these rules exist precisely to avoid YAPAEs from previous versions.

The original text that I’m using as a source for this blog post began as something of a screed: I thought I had found a curious corner to which I was firmly on the “awkward” side of. However my opinion flipped completely about halfway through the analysis that was to form the basis of my argument. I hope by sharing the process of that reasoning with you that I can convince you of just how low the potential awkwardness of this YAPAE is.

So, join me as we explore a ‘curious corner’ of Perl 6, a journey that will take us straight to the source.

set() which is also Set.new() and ().Set which is also Set()


    > Set("wise men" => 3, "star" => 1, "camels" => 9)
    set(wise men, star, camels)

    > ( "wise men" => 3, "star" => 1, "camels" => 9 ).Set
    set(wise men, star, camels)

    > set("wise men" => 3, "star" => 1, "camels" => 2)
    set("wise men" => 3, "star" => 1, "camels" => 2)

    > Set.new("wise men" => 3, "star" => 1, "camels" => 2)
    set("wise men" => 3, "star" => 1, "camels" => 2)

Well, there’s gotta be more than one way to do it, right? This is Perl, after all…

When I first ran into this, it actually upset me a fair bit. I hack in Perl at my $day_job2 and YAPAEs are a part of life when you are bringing new devs into the world of Perl. So I saw that set() and Set() both exist while also both doing different things and bit my fingernails / ranted on #perl6.

If you read that log, you can maybe tell that I’m a bit worked up. Until TimToady stated it simply: composers are not coercers.

Even though I understood the distinction more clearly, I was still a bit unconvinced and went off to investigate.

Composers like Set.new() and set()

Luckily in Rakudo, investigating the internals of the language mainly implies reading how one would implement Perl 6, using Perl 6! All the examples here are taken from their respective files in src/core without modification or truncation.

First we will take a look at some behavior, and then at the code which implements it.


    > set "we wish" => "you", "a" => "merry", "camel" => "ride"
    set("we wish" => "you", "a" => "merry", "camel" => "ride")

That looks exactly like a Perl 6 subroutine invocation: calling set with an argument which is a single List of Pairs. A quick git grep 'sub set' in rakudo.git reveals this declaration:


   #~~( set_operators.pm ) 
   sub set(*@args --> Set) { Set.new(@args) }

It appears to be some syntactic sugar, a sub which takes all of its arguments as a single list (the * prefix is ‘slurpy’) and then passes it to a constructor. I don’t know about you, but that sugar tastes guilt-free to me. Yet our investigation leads us elsewhere.


    #~~( Set.new() ... is in `submethod Build`? )
    submethod BUILD (:%elems) {
        nqp::bindattr(self, Set, '%!elems', %elems);
    }

There’s no .new definition to be found, but what is this submethod business? It sure sounds constructor related. This is a good candidate for looking up in the Perl 6 specification, which tells us that submethods are “infrastructural methods that shouldn’t be inherited by subclasses” (S12). Useful to note is that BUILD is not a phaser, but it shares Huffman coding3 with them: the all-caps is a hint that the code will run automatically.

blessed be the BUILDers

“The bless method automatically calls all appropriate BUILD routines for the current class, which initializes the object in least-derived to most-derived order,” from Synopsis 12: Objects. This is illustrated by the following snippet:


   class Elf {
        submethod BUILD { 
            say "This Elf is busy BUILDing"; 
        }
        method new {
            say "A shiny new Elf";
        }
    }

    role Build { 
        method new { 
            say "Did someone want to BUILD something?";
            self.bless;
        } 
    }

    class Santa is Elf does Build {
       submethod BUILD { 
          say "Even Santa BUILDs";
       }
    }
    Santa.new;
    
    Did someone want to BUILD something?
    This Elf is busy BUILDing
    Even Santa BUILDs

It is the self.bless invocation in Build.new that triggers BUILDALL, which descends the inheritance chain, calling a non-inherited BUILD if it is present. Elf, which does not call self.bless in it’s new method, would only produce the following when constructed:


    > Elf.new
    A shiny new Elf

BUILD is never called here, but it will be if any descendant or role were to mix in a self.bless in a constructor. For this reason, it is generally a good idea to call self.bless when you create a new method, especially if it is in a subclass. Unless, of course, you’ve got other plans for introducing the .bless call.

Back to the investigation

So, the presence of a BUILD in our Set class means that it wants to do something special during initialization. Even though I don’t know NQP, I can guess at what this code might do by asking myself, “what does a set want to do that is so special?”

The presence of the BUILD method is interesting, but as we’ve just seen, it doesn’t do anything without a .new. Just another file over from Set.pm in src/core is Setty.pm. That looks promising, especially if we noticed class Set does Setty in Set.pm and recognized the role declaration syntax.


    #~~( Setty.pm )
    method new(*@args --> Setty) {
        my %e;
        %e{$_.WHICH} = $_ for @args;
        self.bless(:elems(%e));
    }

Well, the %e{$_.WHICH} = $_ for @args line in the new method from is already taking care of one key component of a Set: making a unique list by internally storing each element of the (again slurpy) @args in a hash. That to me implies that the NQP code is taking the other important bit: immutability. Once again we can reach into src/core to see if we can confirm that suspicion. Perl 6 has a mutable form of Set, called SetHash. Let’s check that out.


    #~~( SetHash.pm )
    # No submethod BUILD to be found...

… so it appears that our assumption is correct! Now we know how to implement ‘hard immutability’ for an attribute in a pinch, even if we don’t have a spare second in the heat of that pinch to learn NQP.

Coercers like ().Set and Set()

Coercers are different. They don’t just flatten your lists into a big list, they reduce any Pairs in your list to just the key, or left-hand value, of any given Pair.


    > %( somehow => "special", snow => "flake" ).Set
    set(somehow, snow)

And because Any defines the .Set method, this allows a Set() call with an argument which descends from Any to coerce. If you are curious you can investigate in Synopsis 13: Overloading.


    > Set( (special => "is", as => "does") )
    set(special, as)

Note I didn’t have to turn it into a hash there. The code below shows that it will take all the thingies in the list and extract keys if it can. Otherwise it will “be on its merry,” so to speak.


    #~~( Any.pm )
    method Set()     {     Set.new-from-pairs(self.list) }

    #~~( Set.pm )
    method new-from-pairs(*@pairs --> Setty) {
        my %e;
        for @pairs {
            when Pair {
                %e{.key.WHICH} //= $_.key if .value;
            }
            default {
                %e{.WHICH} //= $_;
            }
        }
        self.bless(:elems(%e));
    }

Et voila! Coercion of a list of Pairs (or a Hash, as in the first example) no longer produces a set of Pairs as it does in the composer, but rather a set of the keys of those Pairs.


    > (special => "is", as => "does", 'a', bag 'of', 
      'snowflakes', 'snowflakes', 'snowflakes').Set
    set(special, as, a, bag(of, snowflakes(3)))

In contrast to earlier versions of Perl, 'a' does not become a key with the value of bag(of, snowflakes(3)). The difference with Perl 6 is that a Pair, denoted by the two terms separated by =>, is it’s very own type — not just a ‘fat comma’ as we have in Perl 5 (just one example of how P5 YAPAEs influenced the design of P6). The behavior is implemented by the when/default behavior: if you are a Pair, we put your key into the set. Otherwise we stuff your object as a whole in, just as we do in the composer.


    > Set( (special => "is", as => "does", 'a' => bag 'of',
      'snowflakes', 'snowflakes', 'snowflakes') )
    set(special, as, a)

Note the extra pair of parentheses when using the Set() form of the coercer. This is because Pairs are used to pass named parameters to subroutines. We can resolve this either by adding another pair of parentheses, as above, or by quoting our keys (which you can see by comparing this last snippet against the very first one in the post).

Conclusion

I hope that this brief, if twisty, tour of composers and coercers in the context of Set has introduced you to a few new dynamics in Perl 6. If nothing else, if it encourages an impulse to pop open src/core to perform your own investigation then I will consider it a success. A merry Christmas, and a jolly curious corner to you all!

Footnotes:

  • 1 Why else would they result in so much cursing?
  • 2 As opposed to $day-job ;)
  • 3 We need a shorthand which represents Huffman Coding. Perhaps we can borrow a character from a logographic language with a similar associated meaning to stand in?