Bonus Xmas – Concurrent HTTP Server implementation and the scripter’s approach

First of all, I want to highlight Jonathan Worthington‘s work with Rakudo Perl6 and IO::Socket::Async. Thanks Jon!

***

I like to make scripts; write well-organized sequences of actions, get results and do things with them.

When I began with Perl6 I discovered a spectacular ecosystem, where I could put my ideas into practice in the way that I like: script manner. One of these ideas was to implement a small HTTP server to play with it. Looking at other projects and modules related to Perl6, HTTP and sockets I discovered that the authors behind were programmers with a great experience with Object-Oriented programming.

Perl6 paradigms

Perl6 supports the three most popular programming paradigms:

• Object-Oriented
• Functional
• Procedural

I think that the Object-Oriented paradigm is fine when you design an application or service that will grow, will do many and varied things and will have many changes. But I don’t like things that grow too much and will have many changes; that’s why I like scripts, for its native procedural approach, because it promote simplicity and effectiveness quickly. I like small (step by step) things that do great things quickly.

The Functional paradigm is awesome in my opinion; you can take a function and use it like a var, among other amazings things.

Perl6 Supplies are like a V12 engine

When I started with Perl6 shortly after I started the translation of perl6intro.com to Spanish language. Looking at the documentation of Perl6 I discovered the great concurrent potential that Perl6 has. The concurrent aspect of Perl6 was more powerful than I thought.

The idea I had of the HTTP server with Perl6 began with the Perl6 Supplies (Asynchronous data stream with multiple subscribers), specifically with the class IO::Socket::Async. All socket management, data transmission and concurrency is practically automatic and easy to understand. It was perfect for making and play with a small concurrent but powerful service.

Based on the examples of the IO::Socket::Async documentation I started to implement a small HTTP server with pseudoCGI support in the mini-http-cgi-server project, and it worked as I expected. As I got what I wanted, I was satisfied and I left this project for a while. I didn’t like things to grow too much.

But then, preparing a talk for the Madrid Perl Workshop 2017 (thanks to Madrid Perl Mongers and Barcelona Perl Mongers guys for the event support), I had enough motivation to do something more practical, something where web front-end coders could do their job well and communicate with the back-end where Perl6 is awaiting. On the one hand, the typical public html static structure, and on the other hand a Perl6 module including several webservices waiting for the web requests from the front-end guys.

Then Wap6 was born (Web App Perl6).

The Wap6 structure

I like the structure for a web application that Wap6 implements:

• public
• webservices

public folder contains the friendly front-end stuff, like static html, javascript, css, etc., that is, the front-end developer space. The webservices folder contains the back-end stuff: a Perl6 module including a function per webservice.

This same folder level contains the solution entry point, a Perl6 script that, among other things like initialization server parameters, contains the mapping between routes and webservices:

my %webservices =
  '/ws1' => ( &ws1, 'html' ),
  '/ws2' => ( &ws2, 'json' )
;

As you can see, not only the routes are mapped to the corresponding webservice, but also specify the return content-type of the webservice (like HMTL or JSON). That is, you type http://domain/ws1 in the web browser and the ws1 function returns the response data with the corresponding content-type as we will see later.

All the routes to the webservices are in %webservices hash and it is passed to the main funcion wap with other useful named params:

wap(:$server-ip, :$server-port, :$default-html, :%webservices);

The core of Wap6

The wap funcion is located out side, in the core lib module that Wap6 use and contains the concurrent and elegant V12 engine:

react {   
  whenever IO::Socket::Async.listen($server-ip,$server-port) -> $conn {
    whenever $conn.Supply(:bin) -> $buf {
      my $response = response(:$buf, :$current-dir, :$default-html, :%webservices);
      $conn.write: $response.encode('UTF-8');
      $conn.close;
    }
  }
}

This is a threes (react – whenever – IO::Socket::Async) reactive, concurrent and asynchronous context. When a transmission arrives from the web client ($conn), it is placed in a new Supply $buf of bin type ($conn.Suply(:bin)), and $buf with other things like the %webservices hash are sent to the response function that runs the HTTP logic. Finally, the return from the response function is written back to the web client.

The response function (located out side, in the core lib too) contains the HTTP parser stuff: it splits the incoming data (the HTTP entity) into headers and body, it performs validations, it takes basic HTTP header information like the method (GET or POST) and the URI (Uniform Resource Identifier), it determines if the requested resource is a webservice (from the webservices folder) or static file (from the public folder), get the data from the resource (from static file or webservice) and returns back to wap function to write the response to the web client, as we have seen before.

The Webservices

The response function, validates $buf and extract the HTTP method from the request header that can be GET or POST (I don’t think that in the future it will support more HTTP methods). Case of GET method it puts the URL params (if any) into $get-params. Case of POST method, it puts the body request into $body.

Then it’s time to check if the web client has requested a webservice. $get-params includes the URI and is extracted with the URI module, finally the result is placed in $path:

given $path {
  when %webservices{"$_"}:exists {
    my ( &ws, $direct-type ) = %webservices{"$_"};
    my $type = content-type(:$direct-type);
    return response-headers(200, $type) ~ &ws(:$get-params, :$body);
  }
  ..
}

If $path exists in the %webservices hash, the client wants a webservice. Then it extracts the corresponding webservice callable function &ws from %webservices hash (yes, I also love Functional paradigm :-) ) and the correspondig content-type. Then it calls the webservice function &ws with the $get-params and the request $body parameters. Finally it returns the HTTP response entity that concatenates:

• The response headers with the status HTTP 200 OK and the given content-type (from the content-type function).
• The webservice output.

The callable webservice &ws can be ws1, located in the Perl6 module from webservices folder:

sub ws1 ( :$get-params, :$body ) is export {
  if $get-params { return 'From ws1: ' ~ $get-params; }
  if $body { return 'From ws1: ' ~ $body; }
}

In this demo context the webservice simply returns the input, that is, the $get-params (when GET) or the $body (when POST).

When the client request a static file

After discarding all the other possibilities, if the client request a static file hosted in the public folder, like html, js, css, etc, then:

given $path {
..
  default {
    my $filepath = "$current-dir/public/$path";
    my $type = content-type(:$filepath);
    return response-headers(200, $type) ~ slurp "$current-dir/public/$path";
  }
}

It returns the response headers including the matched content-type and the requested file contents with slurp.

And that’s all folks! a concurrent web server in the script-procedural manner: Wap6.

Epilogue

I’m happy with the results of Wap6. I don’t pretend that it grows a lot, but I’m always tempted to continue adding more features: SSL support (completed!), session management (in progress), cookies, file uploads, etc.

Perl6 has put on the table a very powerful way to perform concurrent network operations: IO::Socket::Async, a masterpiece. Also, with Perl6 you can mix the Object-Oriented, Procedural and Functional paradigms as you wish. With these capabilities you can design a concurrent asynchronous service and implement it quickly.

If you want something more serious approach with HTTP services and concurrency in the Perl6 ecosystem, take a look at Cro, it represents a great opportunity to establish Perl6 as a powerful entity in the HTTP services space. Jonathan Worthington wrote about it last 9th on this same Advent Calendar.

Meanwhile, I will continue playing with Wap6, in the script manner, contributing with the Perl6 ecosystem and learning from the bests coders in the world, I mean: Perl and Perl6 coders, of course :-)