Day 24 – Make It Snow

Hello again, fellow sixers! Today I’d like to take the opportunity to highlight a little module of mine that has grown up in some significant ways this year. It’s called Terminal::Print and I’m suspecting you might already have a hint of what it can do just from the name. I’ve learned a lot from writing this module and I hope to share a few of my takeaways.

Concurrency is hard

Earlier in the year I decided to finally try to tackle multi-threading in Terminal::Print and… succeeded more or less, but rather miserably. I wrapped the access to the underlying grid (a two-dimensional array of Cell objects) in a react block and had change-cell and print-cell emit their respective actions on a Supply. The react block then handled these actions. Rather slowly, unfortunately.

Yet, there was hope. After jnthn++ fixed a constructor bug in OO::Monitors I was able to remove all the crufty hand-crafted handling code and instead ensure that method calls to the Terminal::Print::Grid object would only run in a single thread at any given time. (This is the class which holds the two-dimensional array mentioned before and was likewise the site of my react block experiment).

Here below are the necessary changes:

- unit class Terminal::Print::Grid;
+ use OO::Monitors;
+ unit monitor Terminal::Print::Grid;

This shift not only improved the readability and robustness of the code, it was significantly faster! Win! To me this is really an amazing dynamic of Perl 6. jnthn’s brilliant, twisted mind can write a meta-programming module that makes it dead simple for me to add concurrency guarantees at a specific layer of my library. My library in turn makes it dead simple to print from multiple threads at once on the screen! It’s whipuptitude enhancers all the the way down!

That said, our example today will not be writing from multiple threads. For some example code that utilizes async, I point you to examples/async.p6 and examples/matrix-ish.p6.

Widget Hero

Terminal::Print is really my first open source library in the sense that it is the first time that I have started my own module from scratch with the specific goal of filling a gap in a given language’s ecosystem. It is also the first time that I am not the sole contributor! I would be remiss not to mention japhb++ in this post, who has contributed a great deal in a relatively short amount of time.

In addition to all the performance related work and the introduction of a span-oriented printing mechanism, japhb’s work on widgets especially deserves its own post! For now let’s just say that it has been a real pleasure to see the codebase grow and extend even as I have been too distracted to do much good. My takeaway here is a new personal milestone in my participation in libre/open software (my first core contributor!) that reinforces all the positive dynamics it can have on a code base.

Oh, and I’ll just leave this here as a teaser of what the widget work has in store for us:

rpg-ui-p6

You can check it out in real-time and read the code at examples/rpg-ui.p6.

Snow on the Golf Course

Now you are probably wondering, where is the darn, snow! Well, here we go! The full code with syntax highlighting is available in examples/snowfall.p6. I will be going through it step by step below.

use Terminal::Print;

class Coord {
    has Int $.x is rw where * <= T.columns = 0;
    has Int $.y is rw where * <= T.rows = 0 ;
}

Here we import Terminal::Print. The library takes the position that when you import it somewhere, you are planning to print to the screen. To this end we export an instantiated Terminal::Print object into the importer’s lexical scope as T. This allows me to immediately start clarifying the x and y boundaries of our coordinate system based on run-time values derived from the current terminal window.

class Snowflake {
    has $.flake = ('❆','❅','❄').roll;
    has $.pos = Coord.new;
}

sub create-flake {
    state @cols = ^T.columns .pick(*); # shuffled
    if +@cols > 0 {
        my $rand-x = @cols.pop;
        my $start-pos = Coord.new: x => $rand-x;
        return Snowflake.new: pos => $start-pos;
    } else {
        @cols = ^T.columns .pick(*);
        return create-flake;
    }
}

Here we create an extremely simple Snowflake class. What is nice here is that we can leverage the default value of the $.flake attribute to always be random at construction time.

Then in create-flake we are composing a way to make sure we have hit every x coordinate as a starting point for the snowfall. Whenever create-flake gets called, we pop a random x coordinate out of the @cols state variable. The state variable enables this cool approach because we can manually fill @cols with a new randomized set of our available x coordinates once it is depleted.

draw( -> $promise {

start {
    my @flakes = create-flake() xx T.columns;
    my @falling;
    
    Promise.at(now + 33).then: { $promise.keep };
    loop {
        # how fast is the snowfall?
        sleep 0.1; 
    
        if (+@flakes) {
            # how heavy is the snowfall?
            my $limit = @flakes > 2 ?? 2            
                                    !! +@flakes;
            # can include 0, but then *cannot* exclude $limit!
            @falling.push: |(@flakes.pop xx (0..$limit).roll);  
        } else {
            @flakes = create-flake() xx T.columns;
        }
    
        for @falling.kv -> $idx, $flake {
            with $flake.pos.y -> $y {
                if $y > 0 {
                    T.print-cell: $flake.pos.x, ($flake.pos.y - 1), ' ';
                }

                if $y < T.rows {
                    T.print-cell: $flake.pos.x, $flake.pos.y, $flake.flake;            
                }

                try {
                    $flake.pos.y += 1;
                    CATCH {
                        # the flake has fallen all the way
                        # remove it and carry on!
                        @falling.splice($idx,1,());
                        .resume;
                    }
                }
            }
        }
    }
}

});

Let’s unpack that a bit, shall we?

So the first thing to explain is draw. This is a handy helper routine that is also imported into the current lexical scope. It takes as its single argument a block which accepts a Promise. The block should include a start block so that keeping the argument promise works as expected. The implementation of draw is shockingly simple.

So draw is really just short-hand for making sure the screen is set up and torn down properly. It leverages promises as (I’m told) a “conv-var” which according to the Promises spec might be an abuse of promises. I’m not very futzed about it, to be honest, since it suits my needs quite well.

This approach also makes it quite easy to create a “time limit” for the snowfall by scheduling a promise to be kept at now + 33 — thirty three seconds from when the loop starts. then we keep the promise and draw shuts down the screen for us. This makes “escape” logic for your screensavers quite easy to implement (note that SIGINT also restores your screen properly. The most basic exit strategy works as expected, too :) ).

The rest is pretty straightforward, though I’d point to the try block as a slightly clever (but not dangerously so) combination of where constraints on Coord‘s attributes and Perl 6’s resumable exceptions.

Make it snow!

And so, sixers, I bid you farewell for today with a little unconditional love from ab5tract’s curious little corner of the universe. Cheers!

snowfall-p6

Day 24 – One Year On

This time of year invites one to look back on things that have been, things that are and things that will be.

Have Been

I was reminded of things that have been when I got my new notebook a few weeks ago. Looking for a good first sticker to put on it, I came across an old ActiveState sticker:

If you don’t know Perl
you don’t know Dick

A sticker from 2000! It struck me that that sticker was as old as Perl 6. Only very few people now remember that a guy called Dick Hardt was actually the CEO of ActiveState at the time. So even though the pun may be lost on most due to the mists of time, the premise still rings true to me: that Perl is more about a state of mind, then about versions. There will always be another version of Perl. Those who don’t know Perl are doomed to re-implement it, poorly. Which, to me, is why so many ideas were borrowed from Perl. And still are!

Are

Where are we now? Is it the moment we know, We know, we know? I don’t think we are at twenty thousand people using Perl 6 just yet. But we’re keeping our fingers crossed. Just in case.

We are now 12 compiler releases after the initial Christmas release of Perl 6. In this year, many, many areas of Rakudo Perl 6 and MoarVM have dramatically improved in speed and stability. Our canary-in-the-coalmine test has dropped from around 14 seconds a year ago to around 5.5 seconds today. A complete spectest run is now about 3 minutes, whereas it was about 4.5 minutes at the beginning of the year, while about 4000 tests were added (from about 50K to 54K). And we now have 757 modules in the Perl 6 ecosystem (aka temporary CPAN for Perl 6 modules), with a few more added every week.

The #perl6 IRC channel has been too busy for me to follow consistently. But if you have a question related to Perl 6 and you want a quick answer, the #perl6 channel is the place to be. You don’t even have to install an IRC client: you can also use a browser to chat, or just follow “live” what is being said.

There are also quite a few useful bots on that channel: they e.g. take care of running a piece of Perl 6 code for you. Or find out at which commit the behaviour of a specific piece of code changed. These are very helpful for the developers of Perl 6, who usually also hang out on the #perl6-dev IRC channel. Which could be you! The past year, at least one contributor was added to the CREDITS every month!

Will Be

The coming year will see at least three Perl 6 books being published. First one will be Think Perl 6 – How To Think Like A Computer Scientist by Laurent Rosenfeld. It is an introduction to programming using Perl 6. But even for those of you with programming experience, it will be a good book to start learning Perl 6. And I can know. Because I’ve already read it :-)

Second one will be Learning Perl 6 by veteran Perl developer and writer brian d foy. It will have the advantage of being written by a seasoned writer going through the newbie experience that most people will have when coming from Perl 5.

The third one will be Perl 6 By Example by Moritz Lenz, which will, as the title already gives away, introduce Perl 6 topics by example.

There’ll be at least two (larger) Perl Conferences apart from many smaller Perl workshops: the The Perl Conference NA on 18-23 June, and the The Perl Conference in Amsterdam on 9-11 August. Where you will meet all sorts of nice people!

And for the rest? Expect a faster, leaner, Perl 6 and MoarVM compiler release on the 3rd Saturday every month. And an update of weekly events in the Perl 6 Weekly on every Monday evening/Tuesday morning (depending on where you live).

Day 23 – Everything is either wrong or less than awesome

Have you ever spent your precious time on submitting a bug report for some project, only to get a response that you’re an idiot and you should f⊄∞÷ off?

Right! Well, perhaps consider spending your time on Perl 6 to see that not every free/open-source project is like this.

In the Perl 6 community, there is a very interesting attitude towards bug reports. Is it something that was defined explicitly early on? Or did it just grow organically? This remains to be a Christmas mystery. But the thing is, if it wasn’t for that, I wouldn’t be willing to submit all the bugs that I submitted over the last year (more than 100). You made me like this.

Every time someone submits a bug report, Perl 6 hackers always try to see if there is something that can done better. Yes, sometimes the bug report is invalid. But even if it is, is there any way to improve the situation? Perhaps a warning could be thrown? Well, if so, then we treat the behavior as LTA (Less Than Awesome), and therefore the bug report is actually valid! We just have to tweak it a little bit, meaning that the ticket will now be repurposed to improve or add the error message, not change the behavior of Perl 6.

The concept of LTA behavior is probably one of the key things that keeps us from rejecting features that may seem to do too little good for the amount of effort required to implement them, but in the end become game changers. Another closely related concept that comes to mind is “Torment the implementors on behalf of the users”.

OK, but what if this behavior is well-defined and is actually valid? In this case, it is still probably our fault. Why did the user get into this situation? Maybe the documentation is not good enough? Very often that is the issue, and we acknowledge that. So in a case of a problem with the documentation, we will usually ask you to submit a bug report for the documentation, but very often we will do it ourselves.

Alright, but what if the documentation for this particular case is in place? Well, maybe the thing is not easily searchable? That could be the reason why the user didn’t find it in the first place. Or maybe we lack some links? Maybe the places that should link to this important bit of information are not doing so? In other words, perhaps there are still ways to improve the docs!

But if not, then yes, we will have to write some tests for this particular case (if there are no tests yet) and reject the ticket. This happens sometimes.

The last bit, even if obvious to some, is still worth mentioning. We do not mark tickets resolved without tests. One reason is that we want roast (which is a Perl 6 spec) to be as full as possible. The other reason is that we don’t want regressions to happen (thanks captain obvious!). As the first version of Perl 6 was released one year ago, we are no longer making any changes that would affect the behavior of your code. However, occasional regressions do happen, but we have found an easy way to deal with those!

If you are not on #perl6 channel very often, you might not know that we have a couple of interesting bots. One of them is bisectable. In short, Bisectable performs a more user-friendly version of git bisect, but instead of building Rakudo on each commit, it has done it before you even asked it to! That is, it has over 5500 rakudo builds, one for every commit done in the last year and a half. This turns the time to run git bisect from minutes to about 10 seconds (Yes, 10 seconds is less than awesome! We are working on speeding it up!). And there are other bots that help us inspect the progress. The most recent one is Statisfiable, here is one of the graphs it can produce.

So if you pop up on #perl6 with a problem that seems to be a regression, we will be able to find the cause in seconds. Fixing the issue will usually take a bit more than that though, but when the problem is significant, it will usually happen in a day or two. Sorry for breaking your code in attempts to make it faster, we will do better next time!

But as you are reading this, perhaps you may be interested in seeing some bug reports? I thought that I’d go through the list of bugs of the last year to show how horribly broken things were, just to motivate the reader to go hunting for bugs. The bad news (oops, good news I mean), it seems that the number of “horrible” bugs is decreasing a bit too fast. Thanks to many Rakudo hackers, things are getting more stable at a very rapid pace.

Anyway, there are still some interesting things I was able to dig up:

  • RT #128804 – this is one of the examples where we attempt to print something better than “syntax error”, but have a problem in the error message itself. This was fixed, and now the error message says Cannot convert string to number: malformed base-35 number in 'li⏏zmat' (indicated by ⏏). Can you spot why this error message is Less Than Awesome?

  • RT #128421 – sometimes we are just wrong for no good reason. Makes you wonder how many other bugs like this are hiding somewhere. Can you find one?

That being said, my favorite bug of all times is RT #127473. Three symbols in the source code causing it to go into an infinite loop printing stuff about QAST nodes. That’s a rather unique issue, don’t you think?

I hope this post gave you a little insight on how we approach bugs, especially if you are not hanging around on #perl6 very often. Is our approach less than awesome? Do you have some ideas for other bots that could help us work with bugs? Leave it in the comments, we would like to know!

Day 22 – Generative Testing

OK! So say you finished writing your code and it’s looking good. Let’s say it’s this incredible sum function:

module Sum {
   sub sum($a, $bis export {
      $a + $b
   }
}

Great, isn’t it?! Let’s use it:

use Sum;
say sum 2, 3; # 5

That worked! We summed the number 2 with the number 3 as you saw. If you carefully read the function you’ll see the variables $a and $b haven’t a type set. If you don’t type a variable it’s, by default, of type Any. 2 and 3 are Ints… Ints are Any. So that’s OK! But do you know what’s Any too? Str (just a example)!

Let’s try using strings?

use Sum;
say sum "bla", "ble";

We got a big error:

Cannot convert string to number: base-10 number must begin with valid digits or '.' in 'bla' (indicated by ⏏)
  in sub sum at sum.p6 line 1
  in block  at sum.p6 line 7

Actually thrown at:
  in sub sum at sum.p6 line 1
  in block  at sum.p6 line 7

Looks like it does not accept Strs… It seems like Any may not be the best type to use in this case.

Worrying about every possible input type for all our functions can prove to demand way too much work, as well as still being prone to human error. Thankfully there’s a module to help us with that! Test::Fuzz is a perl6 module that implements the “technique” of generative testing/fuzz testing.

Generative testing or Fuzz Testing is a technique of generating random/extreme data and using this data to call the function being tested.

Test::Fuzz gets the signature of your functions and decides what generators it should use to test it. After that it runs your functions giving it (100, by default) different arguments and testing if it will break.

To test our function, all that’s required is:

module Sum {
   use Test::Fuzz;
   sub sum($a, $bis export is fuzzed {
      $a + $b
   }
}
multi MAIN(:$fuzz!) {
   run-tests
}

And run:

perl6 Sum.pm6 --fuzz

This case will still show a lot of errors:

Use of uninitialized value of type Thread in numeric context
  in sub sum at Sum.pm6 line 4
Use of uninitialized value of type int in numeric context
  in sub sum at Sum.pm6 line 4
    ok 1 - sum(Thread, int)
Use of uninitialized value of type X::IO::Symlink in numeric context
  in sub sum at Sum.pm6 line 4
    ok 2 - sum(X::IO::Symlink, -3222031972)
Use of uninitialized value of type X::Attribute::Package in numeric context
  in sub sum at Sum.pm6 line 4
    ok 3 - sum(X::Attribute::Package, -9999999999)
Use of uninitialized value of type Routine in numeric context
  in sub sum at Sum.pm6 line 4
    not ok 4 - sum(áéíóú, (Routine))
...

What does that mean?

That means we should use one of the big features of perl6: Gradual typing. $a and $b should have types.

So, let’s modify the function and test again:

module Sum {
   use Test::Fuzz;
   sub sum(Int $a, Int $bis export is fuzzed {
      $a + $b
   }
}
multi MAIN(:$fuzz!) {
   run-tests
}
    ok 1 - sum(-2991774675, 0)
    ok 2 - sum(5471569889, 7905158424)
    ok 3 - sum(8930867907, 5132583935)
    ok 4 - sum(-6390728076, -1)
    ok 5 - sum(-3558165707, 4067089440)
    ok 6 - sum(-8930867907, -5471569889)
    ok 7 - sum(3090653502, -2099633631)
    ok 8 - sum(-2255887318, 1517560219)
    ok 9 - sum(-6085119010, -3942121686)
    ok 10 - sum(-7059342689, 8930867907)
    ok 11 - sum(-2244597851, -6390728076)
    ok 12 - sum(-5948408450, 2244597851)
    ok 13 - sum(0, -5062049498)
    ok 14 - sum(-7229942697, 3090653502)
    not ok 15 - sum((Int), 1)

    # Failed test 'sum((Int), 1)'
    # at site#sources/FB587F3186E6B6BDDB9F5C5F8E73C55195B73C86 (Test::Fuzz) line 62
    # Invocant requires an instance of type Int, but a type object was passed.  Did you forget a .new?
...

A lot of OKs!  \o/

But there’re still some errors… We can’t sum undefined values…

We didn’t say the attributes should be defined (with :D). So Test::Fuzz generated every undefined sub-type of Int that it could find. It uses every generator of a sub-type of Int to generate values. It also works if you use a subset or even if you use a where in your signature. It’ll use a super-type generator and grep the valid values.

So, let’s change it again!

module Sum {
   use Test::Fuzz;
   sub sum(Int:D $a, Int:D $bis export is fuzzed {
      $a + $b
   }
}
multi MAIN(:$fuzz!) {
   run-tests
}
    ok 1 - sum(6023702597, -8270141809)
    ok 2 - sum(-8270141809, -3762529280)
    ok 3 - sum(242796759, -7408209799)
    ok 4 - sum(-5813412117, -5280261945)
    ok 5 - sum(2623325683, 2015644992)
    ok 6 - sum(-696696815, -7039670011)
    ok 7 - sum(1, -4327620877)
    ok 8 - sum(-7712774875, 349132637)
    ok 9 - sum(3956553645, -7039670011)
    ok 10 - sum(-8554836757, 7039670011)
    ok 11 - sum(1170220615, -3)
    ok 12 - sum(-242796759, 2015644992)
    ok 13 - sum(-9558159978, -8442233570)
    ok 14 - sum(-3937367230, 349132637)
    ok 15 - sum(5813412117, 1170220615)
    ok 16 - sum(-7408209799, 6565554452)
    ok 17 - sum(2474679799, -3099404826)
    ok 18 - sum(-5813412117, 9524548586)
    ok 19 - sum(-6770230387, -7408209799)
    ok 20 - sum(-7712774875, -2015644992)
    ok 21 - sum(8442233570, -1)
    ok 22 - sum(-6565554452, 9999999999)
    ok 23 - sum(242796759, 5719635608)
    ok 24 - sum(-7712774875, 7039670011)
    ok 25 - sum(7408209799, -8235752818)
    ok 26 - sum(5719635608, -8518891049)
    ok 27 - sum(8518891049, -242796759)
    ok 28 - sum(-2474679799, 2299757592)
    ok 29 - sum(5356064609, 349132637)
    ok 30 - sum(-3491438968, 3438417115)
    ok 31 - sum(-2299757592, 7580671928)
    ok 32 - sum(-8104597621, -8158438801)
    ok 33 - sum(-2015644992, -3)
    ok 34 - sum(-6023702597, 8104597621)
    ok 35 - sum(2474679799, -2623325683)
    ok 36 - sum(8270141809, 7039670011)
    ok 37 - sum(-1534092807, -8518891049)
    ok 38 - sum(3551099668, 0)
    ok 39 - sum(7039670011, 4327620877)
    ok 40 - sum(9524548586, -8235752818)
    ok 41 - sum(6151880628, 3762529280)
    ok 42 - sum(-8518891049, 349132637)
    ok 43 - sum(7580671928, 9999999999)
    ok 44 - sum(-8235752818, -7645883481)
    ok 45 - sum(6460424228, 9999999999)
    ok 46 - sum(7039670011, -7788162753)
    ok 47 - sum(-9999999999, 5356064609)
    ok 48 - sum(8510706378, -2474679799)
    ok 49 - sum(242796759, -5813412117)
    ok 50 - sum(-3438417115, 9558159978)
    ok 51 - sum(8554836757, -7788162753)
    ok 52 - sum(-9999999999, 3956553645)
    ok 53 - sum(-6460424228, -8442233570)
    ok 54 - sum(7039670011, -7712774875)
    ok 55 - sum(-3956553645, 1577669672)
    ok 56 - sum(0, 9524548586)
    ok 57 - sum(242796759, -6151880628)
    ok 58 - sum(7580671928, 3937367230)
    ok 59 - sum(-8554836757, 7712774875)
    ok 60 - sum(9524548586, 2474679799)
    ok 61 - sum(-7712774875, 2450227203)
    ok 62 - sum(3, 1257247905)
    ok 63 - sum(8270141809, -2015644992)
    ok 64 - sum(242796759, -3937367230)
    ok 65 - sum(6770230387, -6023702597)
    ok 66 - sum(2623325683, -3937367230)
    ok 67 - sum(-5719635608, -7645883481)
    ok 68 - sum(1, 6770230387)
    ok 69 - sum(3937367230, 7712774875)
    ok 70 - sum(6565554452, -5813412117)
    ok 71 - sum(7039670011, -8104597621)
    ok 72 - sum(7645883481, 9558159978)
    ok 73 - sum(-6023702597, 6770230387)
    ok 74 - sum(-3956553645, -7788162753)
    ok 75 - sum(-7712774875, 8518891049)
    ok 76 - sum(-6770230387, 6565554452)
    ok 77 - sum(-8554836757, 5356064609)
    ok 78 - sum(6460424228, 8518891049)
    ok 79 - sum(-3438417115, -9999999999)
    ok 80 - sum(-1577669672, -1257247905)
    ok 81 - sum(-5813412117, -3099404826)
    ok 82 - sum(8158438801, -3551099668)
    ok 83 - sum(-8554836757, 1534092807)
    ok 84 - sum(6565554452, -5719635608)
    ok 85 - sum(-5813412117, -2623325683)
    ok 86 - sum(-8158438801, -3937367230)
    ok 87 - sum(5813412117, -46698532)
    ok 88 - sum(9524548586, -2474679799)
    ok 89 - sum(3762529280, -2474679799)
    ok 90 - sum(7788162753, 9558159978)
    ok 91 - sum(6770230387, -46698532)
    ok 92 - sum(1577669672, 6460424228)
    ok 93 - sum(4327620877, 3762529280)
    ok 94 - sum(-6023702597, -2299757592)
    ok 95 - sum(1257247905, -8518891049)
    ok 96 - sum(-8235752818, -6151880628)
    ok 97 - sum(1577669672, 7408209799)
    ok 98 - sum(349132637, 6770230387)
    ok 99 - sum(-7788162753, 46698532)
    ok 100 - sum(-7408209799, 0)
    1..100
ok 1 - sum

No errors!!!

Currently Test::Fuzz only implement generators for Int and Str, but as I said, it will be used for its super and sub classes. If you want to have generators for your custom class, you just need to implement a “static” method called generate-samples that returns sample instances of your class, infinite number of instances if possible.

Test::Fuzz is under development and isn’t in perl6 ecosystem yet. And we’re needing some help!

EDITED: New now you can only call run-tests()

Day 21 – Show me the data!

Over the years, I have enjoyed using the different data dumpers that Perl5 offers. From the basic Data::Dumper to modules dumping in hexadecimal, JSON, with colors, handling closures, with a GUI, as graphs via dot and many other that fellow module developers have posted on CPAN (https://metacpan.org/search?q=data+dump&search_type=modules).

I always find things easier to understand when I can see data and relationships. The funkiest display belonging to ddd (https://www.gnu.org/software/ddd/) that I happen to fire up now and then just for the fun (in the example showing C data but it works as well with the Perl debugger).

ddd

Many dumpers are geared towards data transformation and data transmission/storage. A few modules specialize in generating output for the end user to read; I have worked on system that generated hundreds of thousands lines of output and it is close to impossible to read dumps generated by, say, Data::Dumper.

When I started using Perl6, I immediately felt the need to dump data structures (mainly because my noob code wasn’t doing what I expected it to do); This led me to port my Perl5 module (https://metacpan.org/pod/Data::TreeDumper  https://github.com/nkh/P6-Data-Dump-Tree) to Perl6. I am now also thinking about porting my HexDump module. I recommend warmly learning Perl6 by porting your modules (if you have any on CPAN), it’s fun, educative, useful for the Perl6 community, and your modules implement a need in a domain that you master leaving you time to concentrate on the Perl6.

My Perl5 module was ripe for a re-write and I wanted to see if and how it would be better if written in Perl6, I was not disappointed.

Perl6 is a big language, it takes time to get the pieces right, for a beginner it may seem daunting, even if one has years of experience, the secret is to take it easy, not give up and listen. Porting a module is the perfect exercise, you can take it easy because you have already done it before, you’re not going to give up because you know you can do it, and you have time to listen to people that have more experience (they also need your work), the Perl6 community has been examplary, helpful, patient, supportive and always present; if you haven visited #perl6 irc channel yet, now is a good time.

.perl

Every object in Perl6 has a ‘perl’ method, it can be used to dump the object and objects under it. The official documentation (https://docs.perl6.org/language/5to6-nutshell#Data%3A%3ADumper) provides a good example.

.gist

Every object also inherits a ‘gist’ method from Mu, the official documentation (https://docs.perl6.org/routine/gist#(Mu)_routine_gist) states: “Returns a string representation of the invocant, optimized for fast recognition by humans.”

dd, the micro dumper

It took me a while to discover this one, I saw that in a post on IRC. You know how it feel when you discover something simple after typing .perl and .gist a few hundred times, bahhh!

https://docs.perl6.org/routine/dd

The three dumpers above are built-in. They are also the fastest way to dump data but as much as their output is welcome, I know that it is possible to present data in a more legible way.

Enter Data::Dump

You can find the module on https://modules.perl6.org/ where all the Perl6 modules are. Perl6 modules link to repositories, Data::Dump source is on https://github.com/tony-o/perl6-data-dump.

Data::dump introduces color, depth limitation, and type specific dumps. The code is a compact hundred lines that is quite easy to understand. This module was quite helpful for a few cases that I had. It also dumps all the methods associated with objects. Unfortunately, it did fail on a few types of objects. Give it a try.

Data::Dump::Tree

Emboldened by the Perl6 community, the fact that I really needed a Dumper for visualization, and the experience from my Perl5 module (mainly the things that I wanted to be done differently) I started working on the module. I had some difficulties at the beginning, I knew nothing about the details of Perl6 and even if there is a resemblance with Perl5, it’s another beast. But I love it, it’s advanced, clean, and well designed, I am grateful for all the efforts that where invested in Perl6.

P6 vs P5 implementation

It’s less than half the size and does as much, which makes it clearer (as much as my newbie code can be considered clean). The old code was one monolithic module with a few long functions, the new code has a better organisation and some functionality was split out to extra modules. It may sound like bit-rot (and it probably is a little) but writing the new code in Perl6 made the changes possible, multi dispatch, traits and other built-in mechanism greatly facilitate the re-factoring.

What does it do that the other modules don’t?

I’ll only talk about a few points here and refer you to the documentation for all the details (https://raw.githubusercontent.com/nkh/P6-Data-Dump-Tree/master/lib/Data/Dump/Tree.pod); also have a look at the examples in the distribution.

The main goal for Data::Dump::Tree is readability, that is achieved with filter, type specific dumpers, colors, and dumper specialization via traits. In the examples directory, you can find JSON_parsed.pl which parses 20 lines of JSON by JSON::Tiny(https://github.com/moritz/json),. I’ll use it as an example below. The parsed data is dumped with .perl,  .gist , Data::Dump, and Data::Dump::Tree

.perl output (500 lines, unusable for any average human, Gods can manage)screenshot_20161219_185724

.gist (400 lines, quite readable, no color and long lines limit the readability a bit). Also note that it looks better here than on my terminal who has problems handling unicode properly.screenshot_20161219_190004

Data::Dump (4200 lines!, removing the methods would probably make it usable)screenshot_20161219_190439

The methods dump does not help.screenshot_20161219_190601

Data::Dump::Tree (100 lines, and you are the judge for readability as I am biased). Of course, Data::Dump::Tree is designed for this specific usage, first it understands Match objects, second it can display only part of the string that are matched, which greatly reduces the noise.
screenshot_20161219_190932

Tweeking output

The options are explained in the documentation but here is a little list
– Defining type specific dumper
screenshot_20161219_185409

– filtering to remove data or add a representation for a data set;  below the data structure is dumped as it is and then filtered (a filter that shows what it is doing).

As filtering happens on the “header” and “footer” is should be easy to make a HTML/DHTML plugin; Althoug bcat (https://rtomayko.github.io/bcat/), when using ASCII glyphs, works fine.

screenshot_20161219_191525
– set the display colors
– change the glyphs
– display address information or not
– use subscripts for indexes
– use ASCII, ANSI, or unicode for the glyphs

Diffs

I tried to implement a diff display with the Perl5 module but failed miserably as it needed architectural changes, The Perl6 version was much easier, in fact, it’s an add-on, a trait, that synchronizes two data dumps. This could be used in tests to show differences between expected and gotten data.

screenshot_20161219_184701
Of course we can eliminate the extra glyphs and the data that is equivalent (I also changed the glyph types to ASCII)screenshot_20161219_185035

From here

Above anything else, I hope many authors will start writing Perl6 modules. And I also hope to see other data dumping modules. As for Data::Dump::Tree, as it gathers more users, I hope to get requests for change, patches, and error reports.

Day 20 – Bridging the gap

Perl 6 arrived last year, after quite a time, I might say! It promised a number of things, which we awaited eagerly. Some it delivered (a solid and sane threading model, for example), some it delivered in perhaps unexpected ways (it was ready by Christmas, after all), but some of them didn’t end up quite how we expected them to. One of those things is Perl 5 interoperability.

It should come as no surprise that CPAN, our legendary repository of modules for Perl 5, with readymade, well documented and thoroughly tested solutions for just about anything you can imagine[1] is quite unlikely to get ported to a completely new language. The bold move to drop backwards compatibility between Perl 5 and Perl 6 was necessary, but it came at a price: the pride of our community, the thing we used to call our language with Perl 5 itself being merely a VM to run it, our bread and butter is now doomed as a mere historical artifact, a legacy we may never live up to, but in the brave, new world of Perl 6 hardly useful anymore. What a loss that would be! Good news is that one of the assumptions of Perl 6 was that it’ll be capable of running Perl 5 code alongside Perl 6 code, loading Perl 5 modules and using them from Perl 6, among other things. Entire power of “good, old” CPAN available at your fingertips, dear reader. Bad news is, however, that this particular functionality we have not entirely get.

The original spec…ulation[2] documents said that you should be able to start a Perl 5 block of code in your Perl 6 code and expect it to work, as so:

grammar Shiny-New-Perl6-Things {
    ...
}

{
    use v5;

    sub good-old-perl5-things($$&) { ... }
}

That requires the Perl 6 compiler to be able to parse Perl 5 (yes, yes, I know; thankfully, the speculation also allows for a “well-behaved subset (…) much like PPI” :)) and execute it, providing interoperability between one and the other. Admirable as it is, it turns out to be much more complicated than it looks to be (and if it does look simple to you, remember to hug a Perl 6 core hacker next time you meet one). It doesn’t mean that there is no effort to make it reality (v5 being one of them, check it out!), but it’s not quite the promised land we were… promised. Yet, I hope! But now is now, and work needs to be done. So what are we left with?

Well, one of the things Perl 6 did deliver marvelously is the foreign function interface, most often used as a way to call C code from Perl 6. So some of us sat down and thought: well, Perl 5 is embeddable and is available to be ran as a C library, so to say. Perl 6 can call functions from C, C can execute Perl 5 code, what’s really stopping us from putting the two and two together? Quite a bit of hard work, but the goal is worthy, and smart, hardworking people is something we have quite in handy in our community. And so, Inline::Perl5 was born.

~~~

The simplest thing are hardly the most exciting ones, but let’s start somewhere:

use Inline::Perl5; # I will be skipping this from now on

my $p5 = Inline::Perl5.new;
$p5.run('print "Hello, older sister!\n"');

Or in a more Perl6-y way (and shorter, so better, right?):

EVAL 'print "Hello, older sister!\n"', :lang<Perl5>;

“But tadzik, how is that better than shelling out and having the external Perl 5 process do some predetermined thing?” Ah, it is better though: we can already drag the results out of the Perl 5 land and get it as a proper, Perl 6 object:

use Inline::Perl5;

my %thing = EVAL 'my %stuff = split(/[=,]/, "foo=bar,baz=beta"); \%stuff', :lang<Perl5>;
for %thing.kv -> $k, $v {
    say "$k => $v"
}

When a little elf saw this, it shouted “woo! Does that mean I can stuff whatever complex magic I want in that EVAL block and I’ll get back a working Perl 6 object?” Why yes, little elf. You can load modules, create objects and pass them around between one Perl and the other, and they’ll work just as you think they will. Check this hot stuff out:

my $mech = EVAL ' use WWW::Mechanize; WWW::Mechanize->new()', :lang<Perl5>;
say $mech.WHAT; # Perl5Object
$mech.get("https://perl6.org");

“Holy Moly!” the little elf could hardly contain itself. This means that the dream of having CPAN still available is still there, still reachable, and as usable as always! Can things get any better?

Hah! Why, I’m glad you asked. Remember how we made EVAL, the Perl 6 mechanism work in the Perl 5-augmented way and produce meaningful results? Syntactic magic is not a Perl 5 exclusive thing, you know. What makes you think we can’t teach use and alike to cooperate with Inline::Perl5 as well?

“Wait, you don’t mean…”

Oh yes, little elf. Yes I do:

use WWW::Mechanize:from<Perl5>;

my $mech = WWW::Mechanize.new;
$mech.get("https://perl6.org");

You don’t always see an elf cry, but when you do, it’s the tears of joy. Just like the ones I saw now, reflected in the comforting, blue-ish[3] light of the computer screen. It’s all there, integrated almost seamlessly, just as the speculations promised. The only price you pay is one use statement and some annotations on the ones that come from The World That Used To Be, and the rest works perfectly fine. All the power of CPAN in all the delicious wrapping paper of the new, shiny Christmas gift of a language. Little elf is a practical little being though, so it immediately started looking for edge cases that’ll prevent its real-world code from being utilized in this way. “I can create objects from Perl 6, call methods from Perl 6, get results back… as long as it can be represented with an object and method calls there’s is no limit to what I can do; in the worst case I can always wrap the good old Perl 5 module in something that plays nice with our limitations…”

I raise my eyebrow, Spock-like. “When would you need to do that?”

“Oh, you know”, started the little elf, “when an exception gets thrown for example.”

We almost couldn’t believe our ears when we heard Santa himself mutter “LOL” from behind his screen. We looked in confusion as he sent us this snippet:

my $mech = WWW::Mechanize.new;
try {
    $mech.get("xmas://perl6.org");
    CATCH {
        default {
            say $_.perl;
        }
    }
}
# Output: X::AdHoc.new(payload => "Error GETing xmas://perl6.org: Protocol scheme 'xmas' is not supported at -e line 0.\n")

The little elf looked almost indignant. “Oh for dancing reindeers, is there anything this thing can’t do!?”

“Like what?” Rudolph asked casually, passing by on its four legs.

“Gee, I don’t think I… ah!” it exclaimed suddenly, finding in its memory a particularly complicated piece of code running the facility at this very moment. “I guess if I need to create objects that are subclasses of existing Perl 5 classes I’m a bit out of luck, am I not?”

“What makes you think that wouldn’t just work as you expect it to?”

“I… heh, I guess I should’ve tried it first. So all the complex things, DBIx::Class, Catalyst, they can all work with this just fine?”

“Been there, done that” muttered Santa from behind his screen again, while the servers hummed peacefully running the now part Perl 5, part Perl 6 production code in the gift factory.

“Well, the future sure does seem bright, does it not?”

“Yes it does”, I replied. “It really does.”

Light years away, a star shone bright, and while it’ll take us a few more days to actually see it above our heads, it is already there, sooner than we thought it would be.

~~~

We may not have gotten the seamless interop we asked for, but for all practical purposes our marvelous Perl 5 legacy is far from gone. It may be almost surprising how much you can already get done with it. Those few battle-tested modules that need glueing together? This nasty script that needs refactoring so badly it may as well end up being a full-blown rewrite? That little project that you dreamed of using Perl 6 for, but never expected to have all the dependencies available (and good, and fast, and proved to work)? It’s time to try it. Your star may already be shining.

[1] I swear I remember seeing a module that existed for the sole purpose of uploading the My Little Pony fanfiction to an appropriate place. If you can help me find it, there’s a prize, why the heck not!

[2] We try very hard to not call it specification :)

[3] Yes, we know about Redshift in the Santa Claus Magical factory, but it’s December, crunch time, we can’t afford to go to bed quite at the time we wish we would be able to. Compromises had to be made.

Day 19 – Fixing Flow

Finding flow while coding is one of the joys of programming.

Encountering simple syntactic bugs, however, can sometimes interrupt flow. A single missing semicolon, for example, can result in a “WAT!?” followed by a “DOH!”

Perl 6 helps you around the code -> run -> fix cycle by identifying the cause and location of a bug and often suggesting a solution. Take this program:

say "hello"
say "world";

When you run it, Perl 6 will suggest what’s wrong, where the problem is, and what to do next …

===SORRY!=== Error while compiling /home/nige/hello-world.pl
Two terms in a row across lines (missing semicolon or comma?)
at /home/nige/hello-world.pl:6
------> say "hello"⏏

That helps to keep things flowing.

Normally, at this point, it’s off to your $EDITOR to manually add the semicolon and the cycle repeats.

What if Perl 6 could suggest a fix and apply it for you?

Introducing perl6fix

Here is the beginnings of a command-line utility called, perl6fix, that takes the hint from Perl 6 and tries to speed up the code -> run -> fix cycle by applying the fix for you.

hello-world

Let’s look at the code.

It needs a handle on bug descriptions found in Perl 6 output.

class Bug {
   has Int        $.line-number;
   has SourceFile $.source-file;
   has Str        $.description;
   has Str        $.pre-context;
}

And a way to describe fixes:

class Fix {

    has $.prompt;
    has $.pattern;
    has $.action;

    method apply ($bug) {
        $!action($bug);
    }
    method applies-to ($bug) {
        return True without $.pattern;
        return $bug.description ~~ $.pattern;
    }
}

And a way to update the source file:

class SourceFile is IO::Path {

    has @!content-lines = self.IO.slurp.lines;

    method append-to-first-matching-line ($append-char, $text-to-match) {
        return unless my $first-matching-index = @!content-lines.first(rx/$text-to-match/, :k);
        @!content-lines[$first-matching-index] ~= $append-char;
        self.save;
    }
    method swap-characters-on-line ($from-chars, $to-chars, $line-number) {
        @!content-lines[$line-number - 1].subst-mutate(/$from-chars/, $to-chars);
        self.save;
    }
    method save {
        self.IO.spurt(@!content-lines.join("\n"));
    }
}

Here is just some of the fixes I encountered while writing this program:

my @fixes = (
    Fix.new(
        prompt  => 'add semicolon',
        pattern => rx/'missing semicolon or comma?'/,
        action  => sub ($bug) {
            $bug.source-file.append-to-first-matching-line(';', $bug.pre-context);
        }
    ),
    Fix.new(
         prompt  => 'add comma',
         pattern => rx/'missing semicolon or comma?'/,
         action  => sub ($bug) {
             $bug.source-file.append-to-first-matching-line(',', $bug.pre-context);
         }
    ),
    Fix.new(
         prompt  => 'convert . to ~',
         pattern => rx/'Unsupported use of . to concatenate strings; in Perl 6 please use ~'/,
         action  => sub ($bug) {
              $bug.source-file.swap-characters-on-line('.', '~', $bug.line-number);
         }
    ),
    Fix.new(
         prompt  => 'convert qr to rx',
         pattern => rx/'Unsupported use of qr for regex quoting; in Perl 6 please use rx'/,
         action  => sub ($bug) {
              $bug.source-file.swap-characters-on-line('qr', 'rx', $bug.line-number);
         }
    ),
    # ADD YOUR OWN FIXES HERE
);

There’s many more potential fixes (I’m just starting).

The perl6fix script wraps perl6, captures STDERR (if there is any), and then looks for a bug report in the output:

sub find-bug ($perl6-command) {

    return unless my $error-output = capture-stderr($perl6-command);

    # show the error
    note($error-output); 

    # re-run the command again - this time grabbing a JSON version of the bug
    # set RAKUDO_EXCEPTIONS_HANDLER env var to JSON 
    return unless my $error-as-json = capture-stderr('RAKUDO_EXCEPTIONS_HANDLER=JSON ' ~ $perl6-command);
    return unless my $bug-description = from-json($error-as-json);

    # just handle these exception types to start with 
    for 'X::Syntax::Confused', 'X::Obsolete' -> $bug-type {
        next unless my $bug = $bug-description{$bug-type}; 
        return Bug.new(
            description => $bug<message>,
            source-file => SourceFile.new($bug<filename>),
            line-number => $bug<line>,
            pre-context => $bug<pre>
        );
    }
}

The next step is to see if there are any fixes for this type of bug:

sub fix-bugs ($perl6-command-line) {

    my $bug = find-bug($perl6-command-line);
 
    unless $bug {
        say 'No bugs found to fix.'; 
        exit;
    }

    # find a potential list of fixes for this type of bug
    my @found-fixes = @fixes.grep(*.applies-to($bug));

    say $bug.description;
    say $bug.source-file.path ~ ' ' ~ $bug.line-number;
    say 'Suggested fixes found: ';

    my $fix-count = 0;

    for @found-fixes -> $fix {
        $fix-count++;
        my $option = ($fix-count == 1)
                   ?? "*[1] " ~ $fix.prompt
                   !! " [$fix-count] " ~ $fix.prompt;
        say ' ' ~ $option;
     }

     my $answer = prompt('apply fix [1]? ') || 1;
     my $fix = @found-fixes[$answer - 1];

     $fix.apply($bug);

     # look for more bugs! until we're done
     fix-bugs($perl6-command-line);
}

With the help of a shell alias you can even run it to fix the previous perl6 command. Like so:

fix-image

Just add an alias to your bash or zsh profile. For example:

alias fix='/home/nige/perl6/perl6fix prev $(fc -ln -1)'

Now it’s your turn

As you can see this is just a start.

You’re welcome to take the full script and evolve your own set of automatic Perl 6 fixes.

You could even adapt the script to apply fixes for Perl 5 or other languages? What about a version using grammars? Or a macro-powered version integrated directly into Perl 6?

Well maybe that last one is something to look forward to next Christmas!

Hope you have a happy Christmas and a flowing, Perl 6 powered 2017!