Christmas is here.

On behalf of the Rakudo development team, I’m proud to announce the Christmas release (December 2015) of Rakudo Perl 6 #94 “коледа”. Rakudo is an implementation of Perl 6 on the Moar Virtual Machine[^1].

This is the Christmas release of Rakudo Perl 6. This version of the compiler targets the v6.c “Christmas” specification of the Perl 6 language. The Perl 6 community has been working toward this release over the last 15 years. Together, we’ve built a language that:

• Retains the core values of Perl: expressiveness, getting the job done, taking influences from natural language, and pushing the boundaries of language design
• Has clean, modern syntax, rooted in familiar constructs but revisiting and revising the things that needed it
• Is truly multi-paradigm, enabling elegant object-oriented, functional, procedural, and concurrent programming
• Serves as a great glue language, allowing for easy calling of C/C++ (using NativeCall) and staying compatible with Perl 5 (via Inline::Perl5).
• Provides composable constructs for working with asynchronous data and parallel computations
• Dramatically reforms and sets a new standard in regex syntax, which scales up to full grammars – powerful enough to parse Perl 6 itself
• Has outstanding Unicode support, with strings working at grapheme level
• Values lexical scoping and encapsulation, enabling easy refactoring
• Is extensible through meta-object programming, user-defined operators, and traits

The tag for this release is “коледа”[^2], a slavic word for an ancient winter festival that has been incorporated into Christmas. We hope you join us in our celebration of getting our Christmas release shipped!

While we are extremely happy to ship an official Perl 6 release, this is not the end of Rakudo’s development. We will continue to ship monthly releases, which will continue to improve performance and our users experience. We’ll also continue our work on the specification, with feedback from the community.

To be clear on that point, this Rakudo release is not considered the primary deliverable for this Christmas; it is the language specification, known as “roast” (Repository Of All Spec Tests), that is considered the primary deliverable. The specification tests that define this 6.c version[^3] of the language are now frozen, and we hope it will be quite some time before we feel obligated to define a 6.d (Diwali) version of the language.

This Rakudo release targets those tests (over 120 thousand of them), and passes them all on at least some architectures when the moon is in the right phase. But Rakudo itself is not frozen. There is still plenty of work ahead for us to improve speed, portability, and stability. Do not expect the level of perfection that you see in established products. This is essentially a .0 release of a compiler. We do not claim an absence of bugs or instabilities. We do not claim the documentation is complete. We do not claim portability to many architectures. We do not claim that all downstream software will work correctly. Think of it as a first kernel release, and now we get to build and port various distributions based around that kernel.

What we do claim is that you now have a stable language specification, and you can enjoy getting some stuff done with Perl 6 without us breaking it every month—as long as you stick to the features that are actually tested in the test suite, that is. Please note that any “feature” you discover that is not tested in the test suite is considered fair game for change without notice.

Have the appropriate amount of fun!

The tarball for this release is available from http://rakudo.org/downloads/rakudo/.

Please note: This announcement is not for the Rakudo Star distribution[^4] — it’s announcing a new release of the compiler and the specification. For the latest Rakudo Star release, see http://rakudo.org/downloads/star/. A Christmas-based version will be available soon.

In addition to being our Christmas release, this is yet another monthly compiler release; Some of the changes that are new in release are outlined below:

New in 2015.12:

• Fixed size and multi-dimensional typed and native arrays
• Greatly overhauled module loading and installation, including handling precompilation at module installation time in Rakudo
• while/until loops can now return lists of values
• We now catch many more kinds of “Useless use of X in sink context”
• A number of convenient Unicode equivalents were introduced
• Superscripts can now be used for integer powers
• Non-digit unicode characters with a numeric value (½ and such) can now be used for that numeric value
• There is a new “approximately equal” operator
• Add support for USAGE argument help
• Provide tau constant (also: τ)
• Can now use channels with supply/react/whenever
• Bool is now a proper enum
• Supply/Supplier improvements
• Use of EVAL now requires a declaration of ‘use MONKEY-SEE-NO-EVAL’
• Make pack and unpack experimental
• Dynamic variables are now visible inside start { … } blocks
• Autoincrements on native ints are now faster than on Int
• The ~~ operator can now chain with other comparisons in many circumstances Various numeric operations now return overflow/underflow failures instead of wrong value
• The :ss, :ii, and :mm options to s/// now all work together

This is only a partial list of the changes in this release. For a more
detailed list, see “docs/ChangeLog”.

The development team thanks all of our contributors and sponsors for making Rakudo Perl possible, as well as those people who worked on the design docs, the Perl 6 test suite, MoarVM and the specification. Additionally, the Pugs, Parrot, and Niecza projects were all instrumental with their contributions to the specification and the community.

The following people contributed to the development of the Christmas release. We’ve gone back through the logs of all the various projects. Thanks to everyone who has worked to make this release happen over the past 15 years. We would also like to thank everyone who submitted bug reports or dropped in on the various forums to discuss things. Finally, we’d like to extend a special thanks to everyone who we accidentally left out of this list.

Gisle Aas, abcxyzp, Chuck Adams, Colin Paul Adams, Rod Adams, C.J. Adams‑Collier, David H. Adler, Chirag Agrawal, Amir E. Aharoni, Bilal Akhtar, Julian Albo, Alekssasho, alexghacker, Paco Alguacil, Brandon S Allbery, Geir Amdal, Markus Amsler, Paul C. Anagnostopoulos, Nikolay Ananiev, anatolyv, andras, Saleem Ansari, Joji Antony, Tomoki Aonuma, Syed Uzair Aqeel, arathorn, Arcterus, Kodi Arfer, Daniel Arbelo Arrocha, ash, Ted Ashton, Arnaud Assad, atroxaper, Ori Avtalion אורי אבטליון, Auzon, Greg Bacon, Ivan Baidakou, Alex Balhatchet, Szabó, Balázs, Amir Livine Bar‑On עמיר ליבנה בר‑און, Luca Barbato, Mattia Barbon, Ann Barcomb, Christian Bartolomäus, Alex “Skud” Bayley, bcmb, Jody Belka, Shachaf Ben‑Kiki, Andrei Benea, benedikth, Zev Benjamin, benmorrow, Kevan Benson, Martin Berends, Anton Berezin, Arthur Bergman, Anders Nor Berle, bestian, Peter Bevan, Mark Biggar, Carlin Bingham, Ævar Arnfjörð Bjarmason, J. David Blackstone, Ronald Blaschke, Ingo Blechschmidt, bloonix, blwood, Kristof Bogaerts, Dan Bolser, Конрад Боровски, Christopher Bottoms, Gonéri Le Bouder, Jos Boumans, Brad Bowman, Matt Boyle, bpetering, H.Merijn Brand, Terrence Brannon, Gwern Branwen, Stig Brautaset, Herbert “lichtkind” Breunung, bri, brian_d_foy, Fernando Brito, Geoffrey Broadwell, Leon Brocard, Benjamin Brodfuehrer, Samuel Bronson, Dan Brook, Nathan C. Brown, Roger Browne, Philippe Bruhat (BooK), David Brunton, Corwin Brust, Klaus Brüssel, Lucas Buchala, buchetc, Christoph Buchetmann, Norbert Buchmuller, Buddha Buck, Alexandre Buisse, Tim Bunce, Bryan Burgers, Sean M. Burke, Matthew Byng‑Maddick, András Bártházi, Jürgen Bömmels, Caelum, Aldo Calpini, Edward Cant, David Cantrell, Carlin, Michael Cartmell, Hezekiah Carty, Nuno ‘smash’ Carvalho, Marcelo Serra Castilhos, Piers Cawley, cdavaz, cdpruden, Gianni Ceccarelli, cfourier, Marc Chantreux, Mitchell N Charity, Oliver Charles, Vasily Chekalkin, Yuan‑Chen Cheng 鄭原真, Daniel Chetlin, Hsin‑Chan Chien 簡信昌, N. Hao Ching, Joshua Choi, Elizabeth Cholet, David Christensen, chuck, cjeris, Nicholas Clark, Steve Clark, Jeff Clites, cmarcelo, cmeyer, Paul Cochrane, Daniel Colascione, Jason Cole, Will “Coke” Coleda, Sylvain Colinet, cono, Tim Conrow, Géraud Continsouzas, Damian Conway, Neil Conway, Stuart Cook, David Corbin, Deven T. Corzine, cosmicnet, Timothy Covell, Beau E. Cox, Simon Cozens, Philip Crow, cspenser, Franck Cuny, Tyler Curtis, David Czech, Daenyth, Dagur, Ritz Daniel, darkwolf, Chris Davaz, David Warring, Justin DeVuyst, Daniel Dehennin, Akim Demaille, Detonite, Lars “daxim” Dieckow 迪拉斯, Matt Diephouse, Bob Diertens, Wendy “woolfy” van Dijk, Jeffrey Dik, John M. Dlugosz, dimid, diotalevi, Hans van Dok, Chris Dolan, Mark Dominus, Bryan Donlan, Andy Dougherty, Dave Doyle, drKreso, dr_df0, dudley, Jonathan Scott Duff, dug, Lee Duhem, Darren Duncan, Andrew Egeler, Havard Eidnes, Nelson Elhage, Fitz Elliott, Alex Elsayed, Jay Emerson, Aankhola Encorporated, ennio, Enveigler, Jon Ericson, Shae Matijs Erisson, Eryq, Mike Eve, Pat Eyler, Aaron Faanes, Kevin Falcone, David Farrell, Angel Faus, Jason Felds, Paul Fenwick, Jose Rodrigo Fernandez, Nelson Ferraz, Adriano Ferreira, João Fernando Ferreira, Chris Fields, Caio Marcelo de Oliveira Filho, Steve Fink, Shlomi “rindolf” Fish שלומי פיש, Mark Leighton Fisher, Scott Fitzenrider, Dudley Flanders, Richard Foley, Vincent Foley, Julian Fondren, Ruben Fonseca, David Formosa, Karl Forner, Solomon Foster, Chaddaï Fouché, Lloyd Fournier, Michael Fowler, Matt Fowles, franck, Austin Frank, Carl Franks, Kent Fredric, Chaim Frenkel, Piotr Fusik, gabriele, John Gabriele, Christoph Gärtner, Martin von Gagern, Felix Gallo, Salvador Ortiz Garcia, Rafaël Garcia‑Suarez, Joshua Gatcomb, Jerry Gay, gcomnz, Jonathan Gentle, iVAN Georgiev, Brian Gernhardt, Bram Geron, Alessandro Ghedini, Imran Ghory, Peter Gibbs, Tanton Gibbs, Brad Gilbert (b2gills), Karl Glazebrook, Nick Glencross, Mark Glines, Flávio S. Glock, Jason Gloudon, Simon Glover, gnuvince, Garrett Goebel, Jeffrey Goff, Mikhael Goikhman, Benjamin Goldberg, Arcady Goldmints‑Orlov, Dimitry Golubovsky, Gerard Goossen, Goplat, Alex Gough, Léo Grange, Chad “Exodist” Granum, Kenneth A Graves, Bruce Gray, Nathan Gray, Mark Grimes, Lucien Grondin, Rolf Grossmann, David Grove, Marcel Grünauer, Daniel Grunblatt, Uri Guttman, gwern, Swaroop C H, Richard Hainsworth, Roger Hale, John “ab5tract” Haltiwanger, Nigel Hamilton, Eric Hanchrow, Sterling Hanenkamp, Ask Bjørn Hansen, Christian “chansen” Hansen, Eirik Berg Hanssen, Samuel Harrington, Trey Harris, John Harrison, Carsten Hartenfels, Richard Hartmann, Kyle Hasselbacher, Austin Hastings, Carl Hayter, Florian Helmberger, Gordon Henriksen, Felix Herrmann, Peter Heslin, Fred Heutte, Jarkko Hietaniemi, Michael H. Hind, Joshua Hoblitt, Zack Hobson, Eric Hodges, Rob Hoelz, Masahiro Honma, Lyle Hopkins, JD Horelick, Jeff Horwitz, Chris Hostetter, Laurens Van Houtven, Jeremy Howard, Yiyi Hu 鹄驿懿, Benedikt Huber, Brad Hughes, Sterling Hughes, Tom Hughes, Tristan Hume, Donald Hunter, Douglas Hunter, Juan Francisco Cantero Hurtado, Kay‑Uwe ‘kiwi’ Hüll, Ran Eliam, Alin Iacob, Ibotty, ibrown, ihrd, Roland Illing, Jan Ingvoldstad, Joshua Isom, isop, Ivan_A_Frey, ivanoff, Akash Manohar J, jafelds, Robert G. Jakabosky, jamesrom, S. A. Janet, jani, Heiko Jansen, Stuart Jansen, Jarrod, Jedai, Chris Jepeway, Chris Jeris, Dagur Valberg Johannsson, Erik Johansen, Paul Johnson, johnspurr, Isaac Jones, Norman Nunley, Jr, Yoshikuni Jujo, Brian S. Julin, Josh Juran, Michal Jurosz, David KOENIG, Prakash Kailasa, Shoichi Kaji, Daniel Kang, Isis Kang, Chia‑Liang Kao 高嘉良, Dmitry Karasik, Luben Karavelov, Amir Karger, Offer Kaye, Bruce Keeler, James E Keenan, Cole Keirsey, Adam Kennedy, Matt Kennedy, Shane Kerr, khairul, Mikhail Khorkov, Krzysztof Kielak, Andres Kievsky, Daehyub Kim, Rob Kinyon, Oleg Kiselyov, OOLLEY kj, Martin Kjeldsen, Thomas “domm” Klausner, Zohar “lumi” Kelrich זהר קלריך/卡卓哈, Damian Miles Knopp, Dan Kogai 小飼弾, Yuval “nothingmuch” Kogman יובל קוג’מן, Tomasz Konojacki, Vadim Konovalov, Nick Kostirya, Matt Kraai, Thomas Kratz, Adrian Kreher, John van Krieken, Matthias Krull, Bradley M. Kuhn, Bob Kuo, Colin Kuskie, Kamil Kułaga, Sage LaTorra, Brent Laabs, laben, Johannes Laire, Markus Laire, Fayland Lam 林道, Mike Lambert, lanny, Leo Lapworth, last.of.the.careless.men, Bart Lateur, Jia‑Hong Lee, Lola Lee, Jonathan Leffler, Tobias Leich, lembark, Mark Lentczner, Moritz A Lenz, Jean‑Louis Leroy, Andy Lester, Jonathan “Duke” Leto, Vladimir Lettiev, Mike Li 李曉光, Stefan Lidman, Yung‑Chung Lin 林永忠, Glenn Linderman, Vladimir Lipsky, Zach Lipton, Stevan Little, Kang‑Min Liu 劉康民, Skip Livingston, David M. Lloyd, Daniel Lo, Peter Lobsinger, Andres Löh, Nick Logan, Eric Lubow, Nolan Lum, Peter Lunicks, LylePerl, Ian Lynagh, lysdexsik, Kiran Kumar M., Jerry MacClure, Noel Maddy, Christopher J. Madsen, Abhijit A. Mahabal, Max Maischein, Peter Makholm, Ujwal Reddy Malipeddi, malon, Christopher Malon, Dagfinn Ilmari Mannsåker, Michael Maraist, Roie Marianer רועי מריאנר, markmont, Simon Marlow, martin, Paolo Martini, Ilya Martynov, Jaume Martí, James Mastros, Michael J. Mathews, Vyacheslav Matjukhin, Tokuhiro Matsuno, mattc, Mike Mattie, Elizabeth “lizmat” Mattijsen, Вячеслав Матюхин, Markus Mayr, Josh McAdams, Martin McCarthy, Mark McConnell, Steven McDougall, John McNamara, Scott McWhirter, mdinger, Olivier “dolmen” Mengué, Kevin Metcalf, Patrick R. Michaud, mimir, mjreed, Tom Moertel, Michael Mol, Paolo Molaro, Shawn M Moore, Brandon Michael Moore, Alexander Moquin, Ion Alexandru Morega, Dino Morelli, Kolia Morev, Zach Morgan, mr_ank, Alex Munroe, muraiki, Paweł Murias, mvuets, Steve Mynott, mzedeler, Carl Mäsak, naesten, Tim Nelson, Ailin Nemui, Ingy döt Net 應吉大聶, David Nicol, Faye Niemeyer, Nigelsandever, Karl Rune Nilsen, Salve J. Nilsen, Per Christian Nodtvedt, Ben Noordhuis, Paweł Nowak, Norman Nunley, Tony O’Dell, יהושע “שי” אוחיון, Clayton O’Neill, Stefan O’Rear, Sean O’Rourke, Matt Oates, Tony Olekshy, Martin Olsen, Dmitriy Olshevskiy, Dave Olszewski, Nelo Onyiah, William Orr, Jon Orwant, Andrei Osipov, Christoph Otto, Pawel Pabian, Walter Pallestrong, Luke Palmer, Bartłomiej Palmowski, Pancake, Paolo, Marton Papp, Andrew Parker, Roman M. Parparov, Anthony Parsons, Mike Pastore, Nova Patch, Timo Paulssen, Tony Payne, Stéphane “cognominal” Payrard, Slava Pechenin, Gael Pegliasco, Stéphane Peiry, Felipe Pena, Nayden Pendov, Wenzel P. P. Peppmeyer, François Perrad, Markus Peter, Ben Petering, Steve Peters, Tim Peterson, Ronny Pfannschmidt, Clinton A. Pierce, Jerrad Pierce, Thilo Planz, plunix, pmakholm, Curtis ‘Ovid’ Poe, Gerd Pokorra, Peter Polacik, Flavio Poletti, Kevin Polulak, John Porter, Ryan W. Porter, Stephen P. Potter, Philip Potter, Adam Preble, premshree, Klāvs Priedītis, Adam Prime, Richard Proctor, Christopher Pruden, Kevin Puetz, Gregor N. Purdy, purl, Hongwen Qiu, Jerome Quelin, quester, Gerhard R., Peter Rabbitson, Florian Ragwitz, raiph, Matt Rajca, Marcus Ramberg, Claudio Ramirez, Prog Rammer, Allison Randal, David Ranvig, Lars Balker Rasmussen, Curtis Rawls, raydiak, Robin Redeker, Ted Reed, Jens Rehsack, Charles Reiss, Gabriele Renzi, Kenneth C. Rich, Jason Richmond, Ryan Richter, Sebastian Riedel, Dennis Rieks, Jens Rieks, Lanny Ripple, John Rizzo, rkhill, Andrew Robbins, Amos Robinson, Jonathan Rockway, Stefano Rodighiero, Andrew Rodland, Lindolfo Rodrigues, Bob Rogers, Dave Rolsky, David Romano, ron, Eric J. Roode, Garret Rooney, Garrett Rooney, David Ross, Andreas Rottmann, Brent Royal‑Gordon, Shmarya Rubenstein, Sam Ruby, Simon Ruderich, Daniel Ruoso, Jake Russo, ruz, Joseph Ryan, Gilbert Röhrbein, Sam S, s1n, sahadev, Patrick Abi Salloum, salty_horse, Chip Salzenberg, Shrivatsan Sampathkumar, Igor Rafael Sanchez‑Puls, Hugo van der Sanden, Thomas Sandlaß, Yun SangHo, sanug, Siddhant Saraf, Sasho, Andrew Savige, John Saylor, Matt Schallert, Bernhard Schmalhofer, Arthur Axel Schmidt, Ronald Schmidt, Michael Schroeder, Steven Schubiger, Steve “thundergnat” Schulze, Andreas Schwab, Randal L. Schwartz, Pepe Schwarz, Frederik Schwarzer, Calvin Schwenzfeier, Michael G. Schwern, Steffen Schwigon, Tom Scola, Ariel Scolnicov, Michael Scott, Peter Scott, Rick Scott, Stefan Seifert, Austin Seipp, Mark Senn, Filip Sergot, Seth Gold (Sgeo), William Shallum, Kris Shannon, shenyute, Aaron Sherman, Mark Shoulson, Ryan Siemens, Ricardo Signes, Hinrik Örn Sigurðsson, Jonathan Sillito, Miroslav Silovic, Steve Simmons, Alberto Manuel Brandao Simoes, John Siracusa, Arne Skjærholt, Barrie Slaymaker, Radek Slupik, Mike Small, Benjamin Smith, Melvin Smith, Tim Smith, Adrian White aka snarkyboojum, Richard Soderberg, SolidState, Vishal Soni, Syloke Soong, Shawn Sorichetti, Tadeusz Sośnierz, sue spence, Cory Spencer, Robert Spier, Michael Stapelberg, Edwin Steiner, stephenpollei, Michael Stevens, Don Stewart, Radu Stoica, Klaas‑Jan Stol, Alek Storm, David Storrs, Mark Stosberg, Jonathan Stowe, Cosimo Streppone, Jonathan Strickland, Pascal Stumpf, Su‑Shee, Sue, Laye Suen, Dan Sugalski, Mark Summerfield, Simon Sun, Cheng‑Lung Sung 宋政隆, Sunnavy, Samuel Sutch, svatsan, svn, Andrew Sweger, sygi, Sebastian Sylvan, Gábor Szabó, Bálint Szilakszi, Marek Šuppa, TOGoS, Arvindh Rajesh Tamilmani, Audrey Tang 唐鳳, Bestian Tang 唐宗浩, Adrian Taylor, Jesse Taylor, Philip Taylor, Kevin Tew, the_dormant, Marcus Thiesen, Adam Thomason, Richard Tibbetts, Carey Tilden, Marcel Timmerman, Leon Timmermans, tkob, John Tobey, Frank Tobin, Bennett Todd, Graham Todd, Leopold Tötsch, Daniel Toma, Nathan Torkington, Timothy Totten, John J. Trammell, Matt S. Trout, Nat Tuck, Adam Turoff, Ben Tyler, ujwal, Bernd Ulmann, Reini Urban, parrot user, uzair, VZ, Ilmari Vacklin, vamped, Wim Vanderbauwhede, vendethiel, David Vergin, Ted Vessenes, Sam Vilain, Cédric Vincent, Jesse Vincent, Jos Visser, John van Vlaanderen, vmax, Jeremy Voorhis, Martin Vorländer, Johan Vromans, Maxim Vuets, Juerd Waalboer, Mariano Wahlmann, Kevin Walker, Gloria Wall, Larry Wall, Lewis Wall, Michal J Wallace, John Paul Wallington, walter, Matthew Walton, Ting Wang, Xinyuan Wang, Andy Wardley, Bryan C. Warnock, wayland, Stephen Weeks, Zhen‑Bang Wei, Nick Wellnhofer, Shu‑Chun Weng, Danny Werner, Brian Wheeler, David E. Wheeler, Dave Whipp, Adrian White, Andrew Whitworth, Bart Wiegmans, Nathan Wiger, Brock Wilcox, Edwin Wiles, Bob Wilkinson, Chris ‘BinGOs’ Williams, Greg Williams, Josh Wilmes, Matthew Wilson, Ashley Winters, Brian Wisti, Dave Woldrich, Helmut Wollmersdorfer, Kevin J. Woolley, Jonathan Worthington, Kuang‑Che Wu 吳光哲, xenu, Liang‑Qi Xie 謝良奇, Xtreak, Gaal Yahas גל יחס, Thomas Yandell, Alvis Yardley, Thomas Yeh 葉志宏, Natan Yellin, yiyihu, Matt Youell, Tony Young, Shen, Yu‑Teh, Ilya Zakharevich, Ruslan Zakirov, Ahmad M. Zawawi, Michael A. D. Zedeler, zengargoyle, zeriod, Agent Zhang 章亦春, Jimmy Zhuo, Ziggy6, Rob Zinkov, Zoffix Znet

If you would like to contribute or find out more information, visit
http://perl6.org, http://rakudo.org/how-to-help, ask on the
perl6-compiler@perl.org mailing list, or ask on IRC #perl6 on freenode.

The next release of Rakudo (#95), is tentatively scheduled for 16 January 2016. A list of the other planned release dates is available in the “docs/release_guide.pod” file.

The development team appreciates feedback! If you’re using Rakudo, do get back to us. Questions, comments, suggestions for improvements, cool discoveries, incredible hacks, or any other feedback — get in touch with us through (the above-mentioned) mailing list or IRC channel. Enjoy!

[^1]: See http://moarvm.org/

[^2]: See https://en.wikipedia.org/wiki/Koliada

[^3]: See https://github.com/perl6/roast/tree/6.c

[^4]: What’s the difference between the Rakudo compiler and the Rakudo Star distribution?

The Rakudo compiler is a compiler for the Perl 6 language. Not much more.

The Rakudo Star distribution is the Rakudo compiler plus a selection of useful Perl 6 modules, a module installer, the most recent incarnation of the “Using Perl 6” book, and other software that can be used with the Rakudo compiler to enhance its utility. Rakudo Star is meant for early adopters who wish to explore what’s possible with Rakudo Perl 6 and provide feedback on what works, what doesn’t, and what else they would like to see included in the distribution.

Day 24 – An Unexpectedly Long-expected Party

Hello. Ting, ting, ting. Attention! … ATTENTION!

Thank you.

Hi, I’m Camelia, if you don’t know me. (Or even if you do.) They told me I had to give a speech at Perl 6’s official Coming Out Party. So here I am. Someone else is doing the roast. I’m just the toast. They told me it had to be a serious speech. Ha, like I’d know how to give a serious speech. Seriously?

Well. Seriously, I’d like to thank you all for coming out today.

Oops, I guess that was a pun. Sorry. Well, no, not really…

But thanks for showing up. This is a big day for Perl 6. She is officially of age now. Well, kinda sorta. She has her driver’s license, anyway. Watch out, world!

[inaudible comment from a table in back]

Oh, should I not have mentioned that? I wasn’t really talking about those fender benders she had while… Well, anyway, let’s move on. I’m sure she’ll be a really good driver. From now on.

Anyway, I have a great deal of empathy for Perl 6, because I am a butterfly. I too had to spend Far Too Long as a chrysalis before I had my own coming out. A literal coming out in my case. Hah, a literal coming out FROM my case, I should say!

Er, oops. I did it again, didn’t I.

Anyway, please be patient with Perl 6. Today we declare her to be a relatively competent being, but she is still just a teenager, y’know. When we’re very young, we’re just the age we are, but when we get to be teenagers, and go through all these hormonal changes, well, we start to oscillate. And the swings just get wider for a while. So when you’re 15, the swing is like plus-or-minus 10 years. One day you’ll act like you’re 25, and the next day like you’re 5 again.

So Perl 6 still has some maturing to do, fershure. That doesn’t mean we don’t love her any less on the days she’s throwing tantrums and driving us crazy. It just means, well, I guess it just means she’s family, and you love family through thick and thin. Because you trust family to turn thin back into thick someday.

And we’re all her extended family, gathered here today. They say it takes a village to raise a child. But never has there been such a child, or such a village! When you get a chance to look at your program tomorrow after your hangover wears off, just look at the list of benefactors in back. There are over 800 names of people who actively contributed to the development of Perl 6, one way or another. And undoubtedly there are a few names missing from the list.

You are a great bunch of people. All of you. Not just the closest members of the family. The family realized long ago that some of the most valuable guidance a growing computer language can get comes from outside the immediate family. It comes from friends and acquaintances who can have more perspective than someone who is always close. That’s why it takes a village.

Maturity is a fractal entity, and operates on many different scales. You all have been a proxy for the wide, wide world that Perl 6 is going out into. And it’s a rough world out there. Perl 6 is ready for some of it, thanks to you.

Of course, she’s still just 15. She does some things really well now. Her communication skills are pretty good, and she is very polite when she can’t understand you. She can carry on several conversations at once. She’s getting pretty good at math, and shows skill in manipulating objects of various sorts. She loves foreign languages, and all those funny characters.

But she’s still a deliberate child, and sometimes seems to be thinking too hard when learning something. That’s okay. She’ll get faster and more efficient over the next several years, as her brain rewires itself into full adulthood. She’ll learn new things, about the world and about herself. But I don’t think her personality will change much, that’s pretty obvious by now.

And that’s because her personality really comes from all of you. You’ve all loved her into existence, and now she’s ready to pass that love on to others who haven’t met her yet.

We all get really excited when a rocket takes off. TimToady tells me about getting up in the wee hours of the morning to watch Mercury, Gemini, and Apollo rockets take off. I’m too young to remember those, but we have our own thrilling developments to track. Me, I was very happy to see a SpaceX rocket stick its landing this week. After several fender benders…

[another inaudible comment]

I’m ignoring you. Be careful. We’ve gotten really, really good at ignoring certain kinds of people. Don’t be one of them.

Really, I feel sorry for the people who are only happy when they have something to be unhappy about.

Anyway, launching Perl 6 into the world is a lot like a rocket launch. A lot of excitement during the countdown, that hold-your-breath moment when you wonder if it’s really going to go up or explode. That’s where we are now. The main engines are lit, the clamps are letting go. It’s all very dramatic, mostly because nothing much seems to be happening right now.

But that’s not what a rocket is about. Drama is not what a rocket wants to do. A rocket wants to go up, faster and faster and faster. It’s not about position. It’s not really even about velocity. It’s about acceleration.

[raises glass]

So, I give you Perl 6. She is free to fly. May she have joy in her existence. May she have joy in discovering the world. May she accelerate as long as she will! Cheers!

Day 23 – macros and secret agents

Let’s talk about 007.

$ perl6 bin/007 -e='say("OH HAI")'
OH HAI

007 is a small language, implemented in Perl 6. Its reason for existing is macros, a topic that has become quite dear to me.

I want to talk a little bit about macros, and what's happened with them in 2015. But first let's just get out of the way any doubts about 007 being a real language. Here, have a Christmas tree.

for [1, 2, 3, 4, 5, 2, 2] -> n {
    my indent = " " x (5 - n);
    my tree = "#" x (2 * n - 1);
    say(indent ~ tree);
}

Which gives this output:

    #
   ###
  #####
 #######
#########
   ###
   ###

If you want to describe 007 real quick, you could say it has a syntax very much like Perl 6, but it borrows things from Python (and JavaScript). For example, there are no sigils. (Awww!) There's no $_, so if you're interested in the elements you're iterating over, you need to declare a loop variable like we did. There's no flattening, no list context, and no conflation between a single item and a list-of-one. (Yay!)

007 does have objects:

my agent = {
    name: "James Bond",
    number: 007
};

say(agent.name);             # James Bond
say(agent.has("number"));    # 1

And, as it happens, all the types of statements, expressions, and other program elements are also expressible in the object system.

my code = quasi {
    say("secret agent!")
};

say(code.statementlist.statements[0]
        .expr.argumentlist.arguments[0]
        .value);             # secret agent!

The quasi there "freezes" the program code into objects, incidentally the exact representation that the 007 compiler deals with. We can then dig into that structure, passing down through a block, a statement, a (function invocation) expression, and finally a string value.

Being able to take this object-oriented view of the program structure is very powerful. It opens up for all kinds of nice manipulations. Perhaps the most cogent thing I've written about such manipulations is a gist about three types of macros. But the way ahead is still a bit uncertain, and 007 is there exactly to scout out that way.

I recently blogged about 007 over at strangelyconsistent, telling a little bit about what's happened with the language in the past year. Here, today, I want to tell about some things that are more directly relevant to Perl 6 macros.

But first...

What are all those Q types, then?

Giving a list of the Q types is still dangerous business, since the list itself is still in mild flux. But with that in mind, let's see what we have.

There's a Q type for each type of statement. Q::Statement::If is a representative example. There's also while loops, for loops, declarations of my variables, constants, subs, macros, etc. Return statements.

A common type of statement is Q::Statement::Expr, which covers anything from a function call to doing some arithmetic. A Q::Statement::Expr is simply a statement which contains a single expression.

So what qualifies as an expression? All the operators: prefix, infix, postfix. Various literal forms: ints, strings. Bigger types of terms like arrays and objects. Identifiers — that is, something like a variable, whether it's being declared or being used. Quasi terms, like we saw above.

Some constructs have some "extra" moving parts that are neither statements nor expressions. For example, blocks have a parameter list with parameters — the parameter list is a Q type, and a parameter is a Q type. Function calls have argument lists. Subs and macros have trait lists. Quasis have "unquotes", a kind of parametric hole in the quasi where a dynamic Qtree can be inserted.

And that's it. For a detailed current list, see Q.pm.

First insight: the magic is in the identifiers

Way back when we started to think about macros in Perl 6, we came to realize that Qtrees are never quite free of their context. You always keep doing variable lookups, calling functions, etc. Because this happens in two vastly different environments (the macro's environment, and the macro user's environment), quite a bit of effort goes to keeping these lookups straight and not producing surprising or unsafe code. This concern is usually referred to as "hygiene", but all we really need to know is that the macro environment is supposed to be able to contain any variable bindings and still not mess up the macro user's environment... and vice versa.

macro moo() {
    my x = "inside macro";
    
    return quasi { say(x) };
}

my x = "outside macro";
moo();    # inside macro

The prevailing solution to this (and the one that I started to code into Rakudo as part of my macros grant) was to achieve hygiene by the clever use of scopes. See this github issue comment for a compelling example of how code blocks would be injected in such a way that the variable lookup would just happen to find the right declaration, even if that means jumping all over the program.

With the recent work of 007, it's evident that the "clever use of scopes" solution won't work all the way. I'm glad I discovered this now, in a toy language implementation, and not after investing lots more time writing up a real solution of this in Rakudo.

The fundamental problem is this: injecting blocks is all good and well in either a statement or an expression. Both of these situations can be made to work. But we also saw in the previous section that there are "extra" Q types which fall outside of the statement/expression hegemony. We don't always think about those, but they are just as important. And they sometimes contain identifiers, which would be unhygienic if there wasn't a solution to cover them. Example: a trait like is looser(infix:<+>). The hygiene consists of infix:<+> being guaranteed to mean what it means in the defining macro environment, not what it means in the using mainline environment.

The new solution is deceptively simple, and will hopefully take us all the way: equip identifiers with a knowledge of what block they were defined in. If we exclude all macro magic, all identifiers will always start their lookup from "here", the position in the code that the runtime is in. (Note that this is already strong enough to account for things like closures, because "here" is exactly the context in which the closure was defined.)

The magic thing about macros is that you will inject a lot of code, and all the identifiers in that code will remember where they were born: in a quasi in a macro somewhere. That's the context in which they will do the lookup, not the "here" of the mainline code. Et voilà, hygiene.

We still allow identifiers to be synthetically created (using object construction syntax) without such a context. This makes them "detached", and the idea is that this will provide a little bit of a safety vent for when people explicitly want to opt out of hygiene. The Common Lisp and Scheme people seem to agree that there are plenty of such cases.

Second insight: there are two ways to create a program

So we know that Qtrees are formed by the parser going over a program, analyzing it, and spitting out Q objects that hang together in neat ways. We know that this process is always "safe", because we trust the parser and the parser is nice.

But in 007 where Q objects are in some sense regular objects, and they can be plugged together in all kinds of way by the user, anything could happen. The user is not necessarily nice — as programmers we take this as a self-evident truth. Someone could try to cram a return statement into a trait! Or drop a parameter list into a constant declaration. Many other horrors could be imagined.

So there have to be rules. Kind of like a HTML DOM tree is not allowed to look any which way it wants. We haven't set down those rules in stone yet in 007, but I suspect we will, and I suspect it will be informative.

(The challenging thing is when we try to combine this idea of restriction with the idea of extending the language. Someone wants to introduce optional parameters. Ok, let's say there's a process for doing that. Now the first obstacle is the rule that says that a parameter consists of just an identifier and nothing else. Once you've negotiated with that rule and convinced it to loosen up, you still have various downstream tools such as linters and code formatters you need to talk to. The problem is not unsolvable as such, just... the right kind of "interesting".)

But it doesn't end with just which nesting relationships — there's a certain sense in which a completely synthetically constructed Qtree isn't yet part of the program as such. For example, it contains variable declarations, but those variable haven't been declared yet. It contains macro calls, but those macros haven't been applied yet. And so on. All of those little things need to happen as the synthetic Qtree is introduced into the bigger program. We currently call this process checking, but the name may change.

Parsing and checking are like two sides of the same coin. Parsing is, in the end, just a way to convince the compiler that a piece of program text is actually a 007 program. Checking is just a way to convince the compiler that a Qtree is actually a 007 program (fragment). We've grown increasingly confident in the idea of checking, partly because we're using it quite heavily in our test suite.

But why stop there? Having two code paths is no fun, and in a sense parsing is just a special case of checking! Or, more exactly, parsing is a very optimized form of checking, where we make use of the textual medium having certain strengths. Why are identifiers limited to the character set of alphanumerics and underscores (and, in Perl 6, hyphens and apostrophes)? Mostly because that's a way for the parser to recognize an identifier. There's nothing at all that prevents us from creating a synthetic Q::Identifier with the name ☺, and there's nothing at all that prevents the checker from processing that correctly and generating a correct program.

It seems to me that the work forwards is to explore the exact relationship between text and Qtrees, between parsing and checking, and to set up sensible rules within which people can write useful macros.

This is exciting work, and will eventually culminate in very nice macros in Perl 6. Expect to hear more about 007 in 2016. See you on the other side!

Day 22 – Perl6 and CPAN

Here’s the short version.

Not yet; but efforts are underway. Stay tuned! News at http://pl6anet.org. And, as always, hit up #perl6 on freenode if you want to talk about it.

Please continue below if you’ve any interest in an overly detailed advent post.

First, I’d like to point out that the imminent Christmas Release is largely not concerned with the topic of this post. I’ll leave it to someone more qualified to qualify the release in more precise terms but suffice it to say its about the Perl6 language and at least one implementation. That does not include non-core ecosystem concerns such as: packaging, distribution, searching, installing, testing services, linters, etc… In the Perl5 world these things are collectively known as “CPAN” and are a huge part of what makes Perl5 useful to many.

The second item I’d like to bring to your attention is that we’ve had an ecosystem solution for quite some time now. Its basically a collection of repos hosted at github (https://github.com/perl6/ecosystem/blob/master/META.list), which can be searched (modules.perl6.org) and installed (panda or zef).
If you want to publish or use Perl6 modules now then this is the way to go. Its probably worth noting that versioning support is not fully implemented yet.

And now onto the future! What should a Perl6 ecosystem be? This is still an open question and an active area of experimentation. Should it be based on shiny things like github and travis? This could probably be made to work by adding versioning support and a few other things to the existing ecosystem. But then there’s the issue of being dependent on entities we don’t control. Should it be built from scratch? I’m aware of one example of that: http://cpan6.org/. Or should it be based on, or even built on top of, Perl5’s CPAN?

I decided a while back I wanted to explore the last of those options.
See http://jdv79.blogspot.com/2015/10/perl6-and-cpan.html and
http://jdv79.blogspot.com/2015/10/perl6-and-cpan-metacpan-status-as-of.html. Those two posts cover it pretty well actually.

Since then all I’ve done is fix Perl6 related bugs in MetaCPAN. Things like syntax highlighting, Pod6 rendering, and of course searching more or less work. One of the better working dists is
http://hack.p6c.org:5001/release/Data-Selector since that’s the one I did the most testing with. And here’s the full listing: http://hack.p6c.org:5001/author/JDV.

Also, just a few days ago, Ranguard decided to help move us forward a bit by basically uploading the ecosystem onto CPAN under the PSIXDISTS user. Unfortunately, or fortunately depending on how you look at it, this led to the discovery of a bug in PAUSE which is not yet fixed (https://github.com/andk/pause/issues/194). It’s probably best to wait until that’s fixed before uploading any Perl6 dists to CPAN.

In summary, we only have the beginnings of PAUSE and MetaCPAN support. Once we get the installers working we’ll have a useful Perl6 CPAN that we can begin to play with.  Or throw out – who knows:) Until then use the ecosystem at http://modules.perl6.org/ for all your Perl6 module needs.

UPDATE: I just noticed the aforementioned PAUSE bug may be fixed as of about an hour ago (~ 1am cet here). Andk++

Day 21 – NativeCall-backs and Beyond C

One of my favorite features in Perl 6 is the NativeCall interface, because it allows gluing virtually any native library into it relatively easily. There have even been efforts to interface with other scripting languages so that you can use their libraries as well.

There have already been a pair of advent posts on NativeCall already, one about the basics in 2010 and one about objectiness in 2011. So this one won’t repeat itself in that regard, and instead be about Native Callbacks and C++ libraries.

Callbacks

While C isn’t quite as good as Perl at passing around functions as data, it does let you pass around pointers to functions to use them as callbacks. It’s used extensively when dealing with event-like stuff, such as signals using signal(2).

In the NativeCall docs, there’s a short quip about callbacks. But they can’t be that easy, can they?

Let’s take the Expat XML library as an example, which we want to use to parse this riveting XML document:

<calendar>
    <advent day="21">
        <topic title="NativeCall Bits and Pieces"/>
    </advent>
</calendar>

The Expat XML parser takes callbacks that are called whenever it finds and opening or closing XML tag. You tell it which callbacks to use with the following function:

XML_SetElementHandler(XML_Parser parser,
                      void (*start)(void *userdata, char *name, char **attrs),
                      void (*end)(void* userdata, char *name));

It associates the given parser with two function pointers to the start and end tag handlers. Turning this into a Perl 6 NativeCall subroutine is straight-forward:

use NativeCall;

sub XML_SetElementHandler(OpaquePointer $parser,
                          &start (OpaquePointer, Str, CArray[Str]),
                          &end   (OpaquePointer, Str))
    is native('expat') { ... }

As you can see, the function pointers turn into arguments with the & sigil, followed by their signature. The space between the name and the signature is required, but you’ll get an awesome error message if you forget.

Now we’ll just define the callbacks to use, they’ll just print an indented tree of opening and closing tag names. We aren’t required to put types and names in the signature, just like in most of Perl 6, so we’ll just leave them out where we can:

my $depth = 0;

sub start-element($, $elem, $)
{
    say "open $elem".indent($depth * 4);
    ++$depth;
}

sub end-element($, $elem)
{
    --$depth;
    say "close $elem".indent($depth * 4);
}

Just wire it up with some regular NativeCallery:

sub XML_ParserCreate(Str --> OpaquePointer)               is native('expat') { ... }
sub XML_ParserFree(OpaquePointer)                         is native('expat') { ... }
sub XML_Parse(OpaquePointer, Buf, int32, int32 --> int32) is native('expat') { ... }

my $xml = q:to/XML/;
    <calendar>
        <advent day="21">
            <topic title="NativeCall Bits and Pieces"/>
        </advent>
    </calendar>
    XML

my $parser = XML_ParserCreate('UTF-8');
XML_SetElementHandler($parser, &start-element, &end-element);

my $buf = $xml.encode('UTF-8');
XML_Parse($parser, $buf, $buf.elems, 1);

XML_ParserFree($parser);

And magically, Expat will call our Perl 6 subroutines that will print the expected output:

open calendar
    open advent
        open topic
        close topic
    close advent
close calendar

So callbacks are pretty easy in the end. You can see a more involved example involving pretty-printing XML here.

C++

Trying to call into a C++ library isn’t as straight-forward as using C, even if you aren’t dealing with objects or anything fancy. Take this simple library we’ll call cpptest, which can holler a string to stdout:

#include <iostream>

void holler(const char* str)
{
    std::cout << str << "!\n";
}

When you try to unsuspectingly call this function with NativeCall:

sub holler(Str) is native('cpptest') { ... }
holler('Hello World');

You get a nasty error message like Cannot locate symbol 'holler' in native library 'cpptest.so'! Why can’t Perl see the function right in front of its face?

Well, C++ allows you to create multiple functions with the same name, but different parameters, kinda like multi in Perl 6. You can’t actually have identical names in a native library though, so the compiler instead mangles the function names into something that includes the argument and return types. Since I compiled the library with g++ -g, I can get the symbols back out of it:

$ nm cpptest.so | grep holler
0000000000000890 T _Z6hollerPKc

So somehow _Z6hollerPKc stands for “a function called holler that takes a const char* and returns void. Alright, so if we now tell NativeCall to use that weird gobbledegook as the function name instead:

sub holler(Str) is native('cpptest') is symbol('_Z6hollerPKc') { ... }

It works, and we get C++ hollering out Hello World!, as expected… if the libary was compiled with g++. The name mangling isn’t standardized in any way, and different compilers do produce different names. In Visual C++ for example, the name would be something like ?holler@@ZAX?BPDXZ instead.

The proper solution is to wrap your function like so:

extern "C"
{
    void holler(const char* str)
    {
        std::cout << str << "!\n";
    }
}

This will export the function name like C would as a non-multi function, which is standardized for all compilers. Now the original Perl 6 program above works correctly and hollers without needing strange symbol names.

You still can’t directly call into classes or objects like this, which you probably would want to do when you’re thinking about NativeCalling into C++, but wrapping the methods works just fine:

#include <vector>

extern "C"
{
    std::vector<int>* intvec_new() { return new std::vector<int>(); }
    void intvec_free(std::vector<int>* vec) { delete v; }
    // etc. pp.
}

There’s a more involved example again.

Some C++ libraries already provide a C wrapper like that, but in other cases you’ll have to write your own. Check out LibraryMake, which can help you compile native code in your Perl 6 modules. There’s also FFI::Platypus::Lang::CPP for Perl 5, which lets you do calls to C++ in a more direct fashion.

Update on 2015-12-22: as tleich points out in the comments, there is an is mangled attribute for mangling C++ function names. So you might be able to call the pure C++ function after all and have NativeCall mangle it for you like your compiler would do – if your compiler is g++ or Microsoft Visual C++:

sub holler(Str) is native('cpptest') is mangled { ... }
holler('Hello World');

It doesn’t seem to be working for me though and fails with a don't know how to mangle symbol error. I’ll amend this post again if I can get it running.

Update on 2015-12-23: the NativeCall API has changed (thanks to jczeus for pointing it out) and now automatically adds a lib prefix to library names. The code changed from is native('libexpat') to is native('expat'). It will also complain that a version should be added to the library name, but I don’t want to weld this code to an exact version of the used libraries.

Perl 6.christmas – have an appropriate amount of fun!

A decade ago Audrey Tang started the Pugs project to implement Perl 6 in Haskell and declared that Perl 6 was optimized for fun: -Ofun.

Game designers know what -Ofun looks like – it’s when players find themselves engrossed in their game, time flies and they experience flow. The art of great game design is balancing the player’s increasing ability with the challenge of the game. Sometimes it’s a bumpy ride!

Flow can happen when programming too: when you’re free from distraction, and the challenge you’re facing is not too easy, not too hard and you’re making positive progress.

Audrey’s invitation to new Perl 6 contributors to have ‘the appropriate amount of fun’ encouraged them to step forward and pick their place on this challenge/ability curve.

The challenge of designing and implementing Perl 6 has presented some scary technical dragons! Fortunately some of the most able programmers stepped forward to hack them down to size.

Over the past decade, it has been a joy to regularly read the #perl6 IRC logs and watch this process unfold. Despite the naysayers, #perl6 has remained a positive, productive place – the propensity for -Ofun is part of the DNA of Perl 6.

Camelia, the Perl 6 butterfly logo, helps sum this up. Camelia includes a reference to a ‘camel’ in her name, however, unlike a camel she doesn’t smell or spit. Flow requires a positive mindset. Camelia is a licence to have -Ofun – not the frivolous, Christmas cracker type, but the deeply productive, enjoyable and enduring type.

The Christmas release is a turning point in the whirlpool of Perl 6 development. The -Ofun will now freely flow outwards to new programmers.

Perl 6, with its emphasis on whipupitude and expressivity, makes it ideally suited to finding flow while programming.

If you haven’t unwrapped Perl 6 yet, this Christmas is the perfect time. You can download Perl 6 here and remember to have an appropriate amount of fun!

Day 19 – Introspection

Perl 6 is an Object-Oriented Programming Language. Well, really, it’s multi-paradigm; but one of the options (that’s used throughout its core) is definitily Object Orientation.

However, as the (current) perl6.org tells us, Perl 6 supports “generics, roles and multiple dispatch”, which are some very nice features, and already covered in other advent calendar posts.

But the one we’re going to take a look at today is the MOP. “MOP” stands for “Meta-Object Protocol”. It means that, instead of objects, classes, etc defining the language; they’re actually a part you can change (or just look at) from the user’s side of things.

Indeed, in Perl 6, you can add methods to a type, remove some, wrap methods, augment classes with more capabilities (OO::Actors and OO::Monitors are two such examples), or you could totally redefine it (and, say, use a Ruby-like object system. Such an example here).

But today, instead, we’re gonna look at the first part: Introspection. Looking at a type after it’s been built, getting to know it, and use these informations.

The module we’re going to build together is based on a need for the Sixcheck module (a QuickCheck-like module): generate some random data for a type, then feed that data to the function we’re testing, and check some post-condition.

So, we write our first version:

my %special-cases{Mu} =
  (Int) => -> { (1..50).pick },
  (Str) => -> { ('a'..'z').pick(50).join('') },
;

sub generate-data(Mu:U \t) {
  %special-cases{t} ?? %special-cases{t}() !! t.new;
}

generate-data(Int);

okay, that’s the first version we wrote. We note a few things:

• We specify the key type for %special-cases. That’s because, by default, the type is Str. Obviously, we don’t want our types to be stringified. What we actually do is specify they’re going to be subtypes of “Mu” (which is at the top of the types “food chain”)
• We put parentheses around Int and Str, to avoid stringification.
• We use the :U in the function’s argument’s type. That means the value has to be undefined. Type objects (as in Int, Str, etc) are undefined, so it serves us well (a different unknown value you’ve probably seen is Nil).
• Type objects really are… Objects, like any other object (more on that later).
  That’s why we can call .new on them, like here. It’s the same as directly calling Int.new, for example (that’s useful for consistency, and autovivification).
• We provide a fallback for Int and Str, because calling Int.new and Str.new (0 and “”) would not give us any randomness in the data we create
• Perl 6 automatically returns the last expression in a function, so we didn’t have to put a return there.

That’s our code to generate the data, fair and square. But we’re gonna need to generate much more than those simple examples.

The least we need to support is classes with properties: we want to look at the list of attributes, generate data for their type, and feed them to the constructor (the only classes we support right now are those with a parameterless constructor).

We need to be able to look inside a class. What we’re going to reach for, in Perl 6 terms, is the Meta-Object Protocol (or MOP for short). First off, let’s define a sample data class for our, huh, blog (the author is sorry for such a lack of imagination).

class Article {
  has Str $.title;
  has Str $.content;
  has Int $.view-count;
}
# we can manually create instances this way:
Article.new(title => "Perl 6 Advent, Day 19",
            content => "Magic!",
            view-count => 0);

But we don’t want to create the article by hand. We want to pass the class Article to our generate-data function, and get an Article back (with random data inside). Let’s go back to our REPL…

> say Article.^attributes;
(Str $!title Str $!content Int $!view-count)
> say Article.^attributes[0].WHAT;
(Attribute)

If you’ve clicked on the MOP link, you shouldn’t be surprised that we get a 3-elements array. If you’re still surprised by the syntax, .^ is the meta-method call. What it means is that a.^b translates to a.HOW.b(a).

If we want to know what’s available to us, we could just ask (removing the anonymous ones):

> Attribute.^methods.grep(*.name ne '<anon>')
(compose apply_handles get_value set_value container readonly package inlined WHY set_why Str gist)
> Attribute.^attributes
Method 'gist' not found for invocant of class 'BOOTSTRAPATTR'

Whoops… Seems like this is a bit too meta. Thankfully, we can use a very nice property of Rakudo: a lot of it is written in Perl 6! To know what’s available to us, we can just look up the source:

#     has str $!name;
...
#     has Mu $!type;

We got the name for the key, and the type to generate the value. Let’s see…

> say Article.^attributes.map(*.name)
($!title $!content $!view-count)
> say Article.^attributes.map(*.type)
((Str) (Str) (Int))

Yep! Seems correct! (If you’re wondering why we get $! (private) twigils back, it’s because $. only means a getter method will be generated. The attribute itself is still private, and accessible in the class).

Now, the only thing we need to build is a loop…

my %args;
for Article.^attributes -> $attr {
 %args{$attr.name.substr(2)} = generate-data($attr.type);
}
say %args.perl;

This is an example of what could be printed:

{:content("muenglhaxrvykfdjzopqbtwisc"), :title("rfpjndgohmasuwkyzebixqtvcl"), :view-count(45)}

You get different results each time you run your code (I don’t think it’ll produce an article worth reading, however…).

Only thing left to do is pass them to Article‘s constructor:

say Article.new(|%args);

(the prefix pipe | allows us to pass %args as named arguments, instead of a single positional argument). Again, you should get something like this printed:

Article.new(title => "kyvphxqmejtuicrbsnfoldgzaw", content => "jqbtcyovxlngpwikdszfmeuahr", view-count => 26)

Yay! We managed to create an Article instance “blindly”, without knowing nothing special about Article at all. Our code can be used to generate data for any constructor that expects its class attributes to be passed. Done!

Before I go, I need to remind you of one things: with great power comes a lot of “WTF”. If you’re only taking a peek like we do here, and gathering information, it should be all fine. But don’t go around and change the internals of Rakudo’s classes, because no one knows what’d happen then :-). As always, have a moderate amount to fun introspecting.

PS: Left as an exercise to the reader is the recursive implementation, move the loop to generate-data, so that we could add a User $.author attribute to Article, and get this one constructed as well. Good luck!

Day 18 – Sized, Typed, Shaped

If you’ve used Perl 5, you probably know PDL (Perl Data Language), the library to manipulate highly efficient data structures.

In Perl 6, we do not (yet) have full-fledged support for PDL (or a PDL-like module), but we do have support for specific features: sized types, shaped variables (and typed arrays).

The first feature, sized types, are low-level types that are composed of a generic low-level type name, and the number of bytes they use. Such an example is int1, int64 (aka int on 64-bit machines), buf8 (a “normal” byte buffer). If you’re interesting in reading the spec, it’s all in S09.

The second feature, shaped variables (still in S09), allow us to specify the fixed size of an array:

my @dwarves[7];

This means the only available indices are 0 to 6 (inclusive). This is an example from the Perl 6’s REPL (Read-Eval-Print-Loop. The input lines start with “>”):

> my @dwarves[7];
[(Any) (Any) (Any) (Any) (Any) (Any) (Any)]
> @dwarves[0] = 'Sleepy';
Sleepy
> @dwarves[7];
Index 7 for dimension 1 out of range (must be 0..6)

The last line of input mentions “dimension 1”. Indeed, Perl 6’s shaped variables can be multi-dimensional (see S09):

> my @ints[4;2];
[[(Any) (Any)] [(Any) (Any)] [(Any) (Any)] [(Any) (Any)]]
> @ints[4;3]
Index 3 for dimension 2 out of range (must be 0..1)
> @ints[4;1]
Index 4 for dimension 1 out of range (must be 0..3)
> @ints[0;0] = 1;
1
> @ints
[[1 (Any)] [(Any) (Any)] [(Any) (Any)] [(Any) (Any)]]

The first output shows us that Perl 6 filled our array with Any at first: this means the whole array is “empty”, and that it can store anything (since Any is a supertype of Int, Str, etc).

We might not want that: our array should contain integers (Int). Instead, we can declare a typed array:

> my Int @a[8] = 0..7;
[0 1 2 3 4 5 6 7]
> @a[8] = 8;
Index 8 for dimension 1 out of range (must be 0..7)
> @a[0].WHAT.say # print the type
(Int)

This is all very useful in itself, but there’s a special property we can benefit from, by using everything presented in this post together: contiguous storage!

Natively-typed shaped arrays in Perl 6 are specced to take contiguous memory, meaning we get a very good memory layout, and use little memory

> my int @a[2;4] = 1..4, 5..8;
[[1 2 3 4] [5 6 7 8]]
> @a[0;0]++;
2
> @a.WHAT.say
(array[int])

Because we know how much memory each element takes, and how many elements we have, we can very easily calculate the array upfront, once and for all – or even calculate the size necessary to store the elements: we know that int, on our 64-bit computers (that’s int64), we need 2*4*8 bytes for our 2*4 array.

Wrapping up; that’s 64 bytes total (…almost), and we get to still write Perl 6! We can use the operations we know and love (even meta/hyper-operators) and al – they’re still arrays – with our cool performance boost. Amazing!

(note about the size: we need to add some overhead for the runtime; on MoarVM, that’s something like 16 bytes for GC header, 16 bytes for the dimensions list. That’s a grand total of 16+16+64=96 bytes. pretty cool!)

Day 17 – A loop that bears repeating

I don’t believe anyone has ever Advent blogged about this topic. But it’s getting hard to tell, what with almost six years of old posts. 😁

This story starts in Perl 5, where we implement a countdown with a simple while loop:

$ perl -wE'my $c = 5; while ($c--) { say $c }'
4
3
2
1
0

And of course, if we want to exit early from the loop, we use the trusty last:

$ perl -wE'my $c = 5; while ($c--) { say $c; last if $c == 2 }'
4
3
2

So far, so good. But notice that $c is decremented before each run of the loop body. Under some circumstances it makes more sense to do it afterwards. In Perl 5, we’d use do ... while:

$ perl -wle'my $c = 5; do { print $c; } while $c--'
5
4
3
2
1
0

Great. Now we just need to combine this with the early exit:

$ perl -wle'my $c = 5; do { print $c; last if $c == 2 } while $c--'
5
4
3
2
Can't "last" outside a loop block at -e line 1.

Eh? Oops.

Experienced Perl people will know what’s going on here. The documentation, in this case perldoc perlsyn, explains it clearly:

The “while” and “until” modifiers have the usual “”while” loop” semantics (conditional evaluated first), except when applied to a “do”-BLOCK […], in which case the block executes once before the conditional is evaluated.

…and then…

Note also that the loop control statements described later will NOT work in this construct, because modifiers don’t take loop labels. Sorry.

In other words, the do ... while construct is a little bit of a cheat in Perl 5, made to work as we expect, but really just a do block with reindeer horns glued on top of its head. The cheat is exposed when we try to next and last from the do block.

perldoc perlsyn goes on to explain how you can mitigate this with more blocks and loop labels, but in a way, the damage is already done. We’ve lost a little bit of that natural belief in goodness, and Santa, and predictable language semantics.

It goes without saying that this is a situation up with which Perl 6 will not put. Let’s see what Perl 6 provides us with.

Instead of painstakingly lining up differences, let’s just first replay our Perl 5 session with the corresponding Perl 6:

$ perl6 -e'my $c = 5; while $c-- { say $c }'
4
3
2
1
0
$ perl6 -e'my $c = 5; while $c-- { say $c; last if $c == 2 }'
4
3
2
$ perl6 -e'my $c = 5; repeat { say $c } while $c--'
5
4
3
2
1
0
$ perl6 -e'my $c = 5; repeat { say $c; last if $c == 2 } while $c--'
5
4
3
2

We’ve gotten rid of a few parentheses, as is usually the case when changing to Perl 6. But the biggest difference is that do has now been replaced by another keyword repeat.

In Perl 5, when we saw the do keyword and a block, we didn’t know if there would come a while or until statement modifier after the block. In Perl 6, when we see the repeat, that’s a promise that this is a repeat ... while or repeat ... until loop. So it’s a real loop in Perl 6, not a fake.

Oh, and the last statement works! As it does in real loops.

---

The story might have ended there, but Perl 6 has a little bit more of a gift to give with this type of loop.

Let’s say we’re waiting for an input that has to have exactly five letters.

sub valid($s) {
    $s ~~ /^ <:Letter> ** 5 $/;
}

This is the type of thing that we’d tend to express with a repeat-style loop, because we have to read the input first, and then find out if we’re done or not:

my $input;
repeat {
    $input = prompt "Input five letters: ";
} until valid($input);

The $input variable has to be declared separately outside of the loop block, because we’re testing it in the until condition outside of the loop block.

Even though we now know to expect a while or until after the block, having that information there makes the end-weight of the loop a bit problematic: we’re delaying some of the most pertinent information until last.

(A similar problem happens in Perl 5 regexes, where a lot of modifiers can show up after the final /, changing the meaning of the whole regex body. And of course, even in spoken and written language, you might have a sentence which goes on and on and just won’t stop, even though it should have long ago, and finally when you think you’ve got the hang of it, it surprises you unduly by ending with altogether the wrong hamster.)

For this reason, Perl 6 allows the following variant of the above repeat loop:

my $input;
repeat until valid($input) {
    $input = prompt "Input five letters: ";
}

That looks nicer!

I hasten to underline that even after moving the condition to the top of the loop like this, the code still has the previous behavior — that is, the condition valid($input) is still evaluated after each iteration of the loop body. (That is, after all, the difference between a repeat until loop and just an until loop.) In other words, we get to place the condition where it’s more prominent and harder to miss, but we retain the expected eval-condition-afterwards semantics.

As a final nice bonus, we can now inline the declaration of $input into the loop condition itself.

repeat until valid(my $input) {
    $input = prompt "Input five letters: ";
}

This clearly shows the difference between order of elaboration (in which variables are declared before they are used) and order of execution (in which statements and expressions evaluate in the order they damn well please).

That’s repeat, folks. Solves a slew of problems, and lets us write nice, idiomatic code.

Day 16 – Yak Shaving for Fun and Profit (or How I Learned to Stop Worry and Love Perl 6)

History

A little over a year and a half ago I decided to write a radio station management application in Perl 5, I won’t bore you with the detailed reasons but it involved hubris and not liking having to modify an existing application that was written in multiple languages that I didn’t enjoy working with.   Most of the required parts were available on CPAN and the requirements were clear, so it was going along, albeit slowly.

Anyway in the early part of this year a couple of things coincided to make me consider that there would be more value in making the application in Perl 6:  the probability of a release before Christmas 2015 tied in with my original estimate that it would take approximately a year to finish the application, the features of Perl 6 that I  was aware of would lead to a nice neat design, and frankly it seemed like a cool idea to get in there with a large, possibly useful, application right as Perl 6 was beginning to enter the mainstream. I largely stopped working on the Perl 5 version and started looking at what I needed to be able to make it in Perl 6.  Of course this would prove to be even more hubristic and deluded than the original idea, but I didn’t know that at the time.

Entering the Yak Farm

It was immediately clear that I was going to have to write a lot more code for myself: at the turn of the year there were approximately 275 distributions in the Perl 6 modules ecosystem, whereas there are somewhere in the region of 30,000 modules on CPAN going back some twenty years. There were bits and pieces that I was going to need: Database access was coming along, there was a RabbitMQ client, people were working on web toolkits so some of the heavy lifting was already being worked on.

Having determined that I was going to have to write some modules it seemed that porting some of my existing CPAN modules might be a good way of getting up to speed and doing something useful into the bargain (though I’m not anticipating using any of them in the larger project.)

I thought I’d start with Linux::Cpuinfo because it was a fairly simple proposition on the face of it and I haven’t been happy with the AUTOLOAD that the Perl 5 version uses since a few months after I wrote it nearly fifteen years ago.

Straight into MOP Land

As it turns out losing the AUTOLOAD in Perl 6 was a fairly simple proposition, infact it could be replaced directly with a method called  “FALLBACK” in the class which gets called with the name of the required method as the first argument thus not requiring the not quite so nice global variable in Perl 5.  But it seemed nicer to take advantage of Perl 6’s Meta Object Protocol (MOP) and build a class based on the fields found in the /proc/cpuinfo on the fly, so I ended up with something like:

multi method cpu_class() {
    if not $!cpu_class.isa(Linux::Cpuinfo::Cpu) {
         my $class_name = 'Linux::Cpuinfo::Cpu::' ~ $!arch.tc;
         $!cpu_class := Metamodel::ClassHOW.new_type(name => $class_name);
         $!cpu_class.^add_parent(Linux::Cpuinfo::Cpu);
         $!cpu_class.^compose;
    }
    $!cpu_class;
}

And then add the fields from the data with:

submethod BUILD(:%!fields ) {
     for %!fields.keys -> $field {
         if not self.can($field) {
             self.^add_method($field, { %!fields{$field} } );
         }
     }
}

In the parent class of the newly made class, works really nicely. This is all going swimmingly, you can construct and manipulate classes using a documented interface without any external dependencies.

Losing the XS

The fact that the Perl 6 NativeCall interface allows you to bind functions defined in a dynamic library directly without requiring any external code didn’t seem to really help with the next couple of modules I chose to look at (Sys::Utmp and Sys::Lastlog) as the majority of the code in the Perl 5 XS files is actually dealing with the differences in various operating systems ideas of the respective interfaces as much as providing the XSUBs to be used by the Perl code.  As it happens this isn’t really so much of a problem as, with all the XSisms stripped out, the code can be compiled and linked to a dynamic library that can be used via NativeCall, even better someone had already made  LibraryMake that makes integrating all this into the standard build mechanisms really quite easy.  Infact it all proved so easy I made both of those modules in a few days rather than just the one that I had intended to do in the first place.

Anyhow by this point it was probably time to start on something that I might need in order to make the radio software, so I settled on Audio::Sndfile – it seemed generally useful and libsndfile is robust and well understood. And this is where the yak shaving really began to kick in. The module itself was really quite easy thanks to NativeCall despite the size of the interface and a subsequently fixed bug in the native arrays, but it occurred to me that automated testing of the module would be somewhat compromised if the dynamic library it depends on is not installed.

Testing Times

In order to facilitate the nicer testing of modules that depend on optionally installed dynamic libraries, I made LibraryCheck, though to be honest it was so easy I was surprised that no-one had made it before.  It exports a single subroutine that returns a boolean to indicate whether the specified library can be loaded or not, so in a test you might do something like:

use LibraryCheck;
if !library-exists('libsndfile') {
    skip "Won't test because no libsndfile", 10;
    exit;
}

As an aside you’ll notice that the extension to the dynamic library isn’t specified because NativeCall works that out for you (be it “.so”, “.dll”, “.dylib” or whatever.)

I’ve actually taken to putting the check in a Build.pm (which is used by panda if present to take user specified actions as part of the build):

class Build is Panda::Builder {
    method build($workdir) {
        if !library-exists('libsndfile') {
            say "Won't build because no libsndfile";
            die "You need to have libsndfile installed";
        }
        True;
     }
}

This has the effect of aborting the build before the tests are attempted, which for automated testing has the benefit of not showing false positives where the tests could not be attempted.

Of course a radio software doesn’t only need to read audio files, it really also should be able to stream audio out to listeners, so I next decided to make Audio::Libshout which binds the streaming source client library for Icecast libshout.  Not only does the testing of this depend on the dynamic library, but also requires the network service supplied by Icecast, so it would be nice to check whether the service was actually available before performing some of the tests.  So to this end I made CheckSocket which does exactly that using what is actually a fairly common pattern in tests for network services. It can be used in a similar fashion to LibraryCheck:

use CheckSocket;
if not check-socket($port, $host) {
    diag "not performing live tests as no icecast server";
    skip-rest "no icecast server";
    exit;
}

I’ve subsequently added it to at least one other author’s tests, it nicely encapsulates a pattern which would otherwise be ten or so lines of boilerplate that would have to be copied and pasted into each test file.

Enter the Gates of Trait

A feature of the implementation of libshout is that it has an initialisation function that returns an opaque “handle” pointer, that gets passed to the other functions of the library, in a sort of object oriented fashion, additionally it provides getter and setter functions for all of the parameters, these would best be modelled as read/write accessors in a Perl 6 class, but there would be a lot of boilerplate code to write these out by hand.  Having looked at a number of similar libraries I concluded that this may be a common pattern, so I wrote AccessorFacade to encapsulate the pattern.

AccessorFacade is implemented as a Perl 6 trait (I won’t explain “trait” here as a previous advent post has already done that,)  it allowed me to turn:

     sub shout_set_host(Shout, Str) returns int32 is native('libshout') { * }
     sub shout_get_host(Shout) returns Str is native('libshout') { * }

     method host() is rw {
         Proxy.new(
                    FETCH => sub ($) {
                                       shout_get_host(self);
                    },
                    STORE => sub ($, $host is copy ) {
                        explicitly-manage($host);
                        shout_set_host(self, $host);
                    }
         );
     }

Of which there may  be over a dozen or so, into:

sub shout_set_host(Shout, Str) returns int32 is native('libshout') { * } 
sub shout_get_host(Shout) returns Str is native('libshout') { * }

method host() is rw is accessor-facade(&shout_set_host, &shout_get_host) { }

Thus saving tens of lines of boilerplate, copy and paste mistakes and simplified testing. It probably took me longer to write AccessorFacade nicely after I had worked out how to do traits, than I ended up doing for Audio::Libshout. Which is a result, as next up I decided I needed to write Audio::Encode::LameMP3 in order to stream audio data that wasn’t already MP3 encoded, and it transpired that the mp3lame library also used the getter/setter pattern that AccessorFacade targetted, having that library enabled me to finish it even quicker than anticipated.

Up to my Ears in Yaks

Digital audio data is typically represented by large arrays of  numbers and with the native bindings there is a lot of creating native CArrays of the correct size, copying Perl arrays to native arrays and copying native arrays to Perl arrays and so forth, this was clearly a generally useful pattern so I created NativeHelpers::Array to collect all the common use cases, refactored all the audio modules to use the subroutines it exports and found myself in the position of having written more modules to help me make the modules that I wanted to write than the modules I actually wanted to write.  So in order to get things in balance I wrote Audio::Convert::Samplerate that used some of the helpers above.

Around this time I decided that I probably needed to concentrate on some application infrastructure requirements rather than domain specific things if I wanted to get anywhere with the original plan and started on a logging framework that I had been thinking about for a while.  I immediately concluded that I needed a singleton logger object so I wrote Staticish.

Staticish is implemented as a class trait that basically does two things: it adds a role to the class that gives it a singleton constructor that will always only return the same object (which is created the first time it is called,) and applies a role to the classes MetaClass (the .HOW,) which will cause the methods (and public accessors) of the class to be wrapped such that if the method is called on the type object of the class the singleton object will be obtained from the constructor and it will be called on that instead, so you can do something like:

use Staticish;

class Foo is Static {
    has Str $.bar is rw;
}

Foo.bar = "There you go";
say Foo.bar; # > "There you go";

It does exactly what I need it to, but I still haven’t finished that logging framework.

All at Sea with JSON

Earlier in the year I had started working on a Couch DB interface module with a view to using a document rather than a relational database in the application which is part of the reason I had been helping to make the HTTP client modules do some of the things that were needed, but another part, for me at least, was the ability to round-trip a Perl 6 class marshalled to JSON and back again, or vice versa. Half of that already existed with JSON::Unmarshal so I proceeded to make JSON::Marshal to do the opposite and then JSON::Class which provides a role such that a class has  to-json and from-json methods, all so good so far and you can do:

 use JSON::Class;

 class Something does JSON::Class {
     has Str $.foo;
 }

 my Something $something = Something.from-json('{ "foo" : "stuff" }');
 my Str $json = $something.to-json(); # -> '{ "foo" : "stuff" }'

But it needed some real world application to test it in, and fortunately this presented itself in the Perl 6 ecosystem itself: the META6.json that is crucial to the proper installation of a module. It is quite easy to mis-type if you are manually creating JSON, so META6 which can read and write the META6 files using JSON::Class and Test::META which can provide a sanity test of the meta file seemed like a good idea. Almost inevitably JSON::Unmarshal and JSON::Marshal needed to be revisited to allow custom marshallers/un-marshallers to be defined, (using traits, natch,) for certain attributes.

I had actually already started making JSON::Infer a year ago in Perl 5 for reasons that I could make an entire post about, but having already started down the JSON path it seemed easy to finish in Perl 6 and use JSON::Class in the created classes to allow the creation of JSON web service API classes easily: unfortunately JSON::Class (or rather its depencies) didn’t even survive the encounter with the test data, so I made JSON::Name so JSON object attribute who’s name wasn’t a valid Perl 6 identifier could be marshalled or unmarshalled correctly.  All fixed up this gave me the impetus to finish the WebService::Soundcloud which I had been playing with for a year or so.  I still haven’t finished the CouchDB interface.

Nowhere near to Land

So now toward the end of the year, there are 478 modules in the ecosystem and somehow I have made 29 of them and I haven’t even started to make the application that I set out to make at the beginning of the year, but I’m happy with that as hopefully I’ve made some modules that may be useful to someone else, I’ve become a confident Perl 6 programmer and the things I’ve learned have helped improve the documentation and possibly some of the code. I will finish the radio software in Perl 6 and I still know I have a lot of software to write but it becomes easier every day.  So if you’re thinking of making an application in Perl 6 yourself enjoy the journey more than you may be frustrated by not reaching the destination when you expected as you are probably travelling a path along which few have been before.