Day 15 – 2015 The Year of The Great List Refactor

The Great List Refactor (GLR) was probably one of the most tricky and problematic changes in terms of design and implementation that Perl 6 has undergone recently.

It’s also fairly hard to explain! But hopefully some historical background might help with this. There was much GLR discussion at the 2014 Austrian Perl Workshop which was written up by Patrick Michaud on his blog.

The problems the GLR was intended to address were those of performance and consistency in the way lists and related types operated. It was also clear that changing such basic data types would be painful.

Traditionally Perl 5 flattens such that

% perl -dE 1 -MData::Dumper

[...snip...]

DB<1> @foos=((1,2),3)

DB<2> say Dumper \@foos
$VAR1 = [
         1,
         2,
         3
        ];

and initially much Perl 6 behaviour resembled this but by the end of last year there had been increased use of non-flattening behaviour which preserves the original data structure.

However this was not consistent with many edge cases and there was use of a data type Parcel which was heavily used internally in Rakudo but by then regarded as a Bad Idea – mainly for reasons of performance.

A draft twin of the design document covering lists “Synopsis 7” was imported by Patrick in June 2015 and became the official S07 the following month.

August was a very busy month for the GLR.  Jonathan Worthington experimented with a single gist based GLR code fragment which then became a separate GLR git branch under the Rakudo repository.  Many people worked on this branch with Stefan Seifert (nine) being particularly productive.  For a while the IRC channel had two bots camelia and GLRelia (the latter tracking the GLR branch) so people could quickly try and compare the behaviour of code under the old and new systems.  There were a huge number of changes and software such as panda had to be patched to remain functional.

The Parcel data type became List (which was immutable and used round brackets) and Array was a mutable subclass of List which used square brackets.  There was no longer implicit list flattening for arrays.  Simple arrays could be flattened with the .flat method.

Pre GLR

> my @array = 1,(2,3),4

1 2 3 4

> @array.elems

4

Post GLR

my @array = 1,(2,3),4

[1 (2 3) 4]

> @array.elems

3

> my @array = (1,(2,3),4).flat

[1 2 3 4]

It was also possible to “Slip” a list into an array

> my @a = 1, (2, 3).Slip, 4

[1 2 3 4]

Also

> my @a = 1, |(2, 3), 4
[1 2 3 4]

Sequences (to be consumed once only) were introduced

> my $grep = (1..4).grep(*>2)
(3 4)
> $grep.^name
Seq

and the .cache method can be used to prevent consumption of the Seq

The Single Argument Rule

The arguments passed to iterators such as the for loop obey “the single arg rule” which is that they are treated as a single arg even if they appear multiple arguments at first glance. Generally this makes for behave as the programmer would expect apart from one Gotcha example with a trailing comma.

my @a=1,2,3;

my ($i,$j);
for (@a) {
    $i++;
}

for (@a,) { # this is actually a single element list!
    $j++;
}

say :$i.gist; # => 3
say :$j.gist; # => 1

S07 was then rewritten by Jonathan Worthington in September 2015. The net result was by then S07 had under gone many changes eg. Parcel was removed, reintroduced and finally removed again!

https://github.com/perl6/specs/blob/master/S07-lists.pod

For such a radical change to Perl 6 the GLR went remarkably smoothly within Rakudo itself.  But of course much existing Perl 6 outside of Rakudo in the ecosystem and other places had to be rewritten in order to continue to work. An example is "perl6-examples" which seem to use lists and arrays particularly heavily.

For many months the GLR had been the Bugbear of Perl 6. A bugbear which has now happily departed.