Thursday, October 20, 2011

Catalyst::Action::Serializer::Plist::Binary

Last weekend, at DrupalCamp WNY, I attended a session entitled Semi-homemade Web Services in Drupal 7 given by Brian Fending. One of the examples was a RESTful API which provided data to an IPad app serialized as binary plist. This struck me as one of the better ideas I've heard lately. Property lists are native to iOS, and the binary format is far more space-efficient than the XML format. If it is at least as efficient as JSON, and some client-end processing can be reduced or avoided, it could be a big win.

Currently, the interfaces I design are being implemented in Catalyst. Since there is no Catalyst plist serializer available on CPAN, I decided to write one (two, actually - I included the XML-based format for completeness). Fortunately, there are plist modules on CPAN - Data::Plist::XMLWriter and Data::Plist::BinaryWriter. Creating the serializer actions requires just a little glue.

Here's the binary serializer action:
package Catalyst::Action::Serialize::Plist::Binary;
use parent 'Catalyst::Action';
use Data::Plist::BinaryWriter;

sub execute {
    my ($self, $controller, $c) = @_;
    $c->res->content_type('application/x-plist');
    $c->res->output(Data::Plist::BinaryWriter->new->write($c->stash->{$controller->config->{'stash_key'} || 'rest' }));
}

1;
And here's the XML serializer action:
package Catalyst::Action::Serialize::Plist;
use parent 'Catalyst::Action';
use Data::Plist::XMLWriter;

sub execute {
    my ($self, $controller, $c) = @_;
    $c->res->content_type('text/plist');
    $c->res->output(Data::Plist::XMLWriter->new->write($c->stash->{$controller->config->{'stash_key'} || 'rest' }));
}

1;

Now to wire it up.

In our Root controller:
sub end : ActionClass('Serialize') {}

In a REST controller:

To add XML plist serialization:
__PACKAGE__->config->{map}{'text/plist'} = 'Plist';
To add binary plist serialization:
__PACKAGE__->config->{map}{'application/x-plist'} = 'Plist::Binary';

In the client application, set the Content-Type header, content-type query parameter or Accept header appropriately, as described in Catalyst::Controller::REST, and voila.

Disclaimer: I whacked these together in a few minutes last night and have not tested them extensively. As time permits, I'll test them systematically and productionalize them.

Tuesday, September 6, 2011

YAAACCC

(Yet Another Argument Against Complete Code Coverage)

It's been said many times, but still bears repeating, that 100% code coverage in unit tests is not a good goal. If you don't already believe that, I probably won't convince you. I'm not going to rehash all the reasons. Look around, you can find lots of eloquent arguments already in print.

But I will give you one that is perhaps less obvious and less frequently cited than some, with a concrete example: code coverage software is just that, software. It is as unlikely to be completely bug-free as any other software.

Case in point: Devel::Cover is a nice package. I really like the color-coded HTML output annotating the code, and showing various coverage statistics, with drill-down. But at work, I frequently encountered low Condition coverage.

I was writing a lot of code that looked something like this:

sub twiddle { 
  my ($self, $c, $user, $device, $knob) = @_; 
  $user = decode_param($user) || return $self->error(400); 
  $device = decode_param($device) || return $self->error(400); 
  $knob = decode_param($knob) || return $self->error(400); 
  return $self->error(401) if !authenticated($user); 
  return $self->error(401) if !authorized($user, $device); 
  my $model = $c->model('Twiddle') || return $self->error(500); 
  return $model->twiddle($device, $knob); 
}
Under Condition Coverage, for a line like $user = decode_param($user) || return $self->error(400); I would get truth tables that looked like:






Even though I had run tests for both success and failure of decode_param($user), I had only 67% condition coverage, evidently because Devel::Cover didn't understand that it couldn't evaluate the truth or falsity of the second clause of ||, since return takes execution out of the current scope.

While sometimes examining coverage output can help improve code, in a case like this, modifying the code to make the coverage tool happy would just be stupid. And, as it turns out, short-lived.

I was writing up this example to present at the Catalyst Developer's Meetup, creating the sample code above and the tests and coverage to use in my presentation, when I discovered that behavior of Devel::Cover had changed. The same code and tests in my home environment give this result:





Turns out that the behavior I described above was fixed between Devel::Cover Version 0.65, and Devel::Cover Version 0.73. This doesn't disprove my point. Rather, it reinforces it. Don't blindly assume that your coverage tool is perfect, or code to its idiosyncrasies.

Code coverage should not be used as a gating factor, or to satisfy bean-counters. It is a useful tool for the developer to ensure that what should be tested is, and nothing more.

Wednesday, August 31, 2011

Kwalitee is Job #1

I've been working on a presentation on unit testing for an upcoming Catalyst Developer's Meetup, and on some real-world unit tests for a REST interface. Eventually, I plan to post the whole presentation on slideshare, but there are a couple things worth mentioning here first.

I was flailing around on the 'net the other day, looking for some ideas on better approaches to model testing, when I ran across something unusual - a technical book worth the asking price -  Perl Testing: A Developer's Notebook.
It's a nice how-to kind of book, and well organized, so it would be great for someone new to the subject. But it also contains a few nuggets that I hadn't stumbled on elsewhere yet.

Here's one of those nuggets: Kwalitee

Open-source software is great. I've been using it for a long time. But the quality varies greatly. Sometimes, even when the software itself is great, the packaging really sucks. Installers are missing, broken, or just not intuitive. Ditto for documentation.

Perl packages from CPAN however, are generally packaged up pretty well. This implies nothing about the quality or usefulness of the software itself, but aside from the occasional broken dependency chain, getting software installed from CPAN is relatively uniform and painless, and there's at least some documentation.

This is apparently due, at least in part to CPANTS, the CPAN Testing Service, and its definition of Kwalitee, a series of metrics that check packages for the existence of things like tests and documentation (and many more - read the docs).

Thanks to Test::Kwalitee, it's easy to add those checks to your perl packages. Simply add this test to your t directory:

use Test::More;
eval { require Test::Kwalitee; Test::Kwalitee->import() };
plan( skip_all => 'Test::Kwalitee not installed; skipping' ) if $@;

Run the test:

prove --lib --verbose t/kwalitee.t
t/kwalitee.t ..
1..13
ok 1 - extractable
ok 2 - has_readme
ok 3 - has_manifest
ok 4 - has_meta_yml
ok 5 - has_buildtool
ok 6 - has_changelog
ok 7 - no_symlinks
ok 8 - has_tests
ok 9 - proper_libs
ok 10 - no_pod_errors
ok 11 - use_strict
ok 12 - has_test_pod
ok 13 - has_test_pod_coverage
ok
All tests successful.
Files=1, Tests=13, 1 wallclock secs ( 0.03 usr 0.00 sys + 0.53 cusr 0.04 csys = 0.60 CPU)
Result: PASS

Address any issues found, and you too have achieved Kwalitee.

Tuesday, July 26, 2011

JSONP - give it a REST!

The limitations of JSONP make it especially unsuited for use with RESTful web services
  1.  limited to GET
  2.  can't handle HTTP status <> 200
  3.  size limitation on GET querystring


And yet, many web applications have a need to access REST-style ( I like to refer to ours as RESTive ) web services cross-origin. JSONP is often used for this. I'm sure there are cases where a developer's hands are tied, but often this is more likely an example of the hammer and nails phenomenon.

In any case, there's no reason to let it infect your server-side code.

I've seen a number of schemes to support JSONP by passing parameters to tell the server that in this case GET really means POST, or PUT or DELETE. Heinous! Don't get me wrong, I'm not some sort of REST zealot. It's a useful fiction at best. But adding unnecessary code is *always* bad.

All you need is a little varnish to overcome (1) and (2). I can't help with (3), sorry.
(It's actually a deal-breaker for us, with some APIs requiring large amounts of POST data.)

First, we need some indicator that the request is JSONP.  In this example I chose to prepend /jsonp/<desired method> to the request URL. Next, since we're going to rewrite the URL, we need a new indicator to remember that it was a JSONP request. I chose the custom header req.http.x-rest-method. Then we need to remove the prepended string from the URL to get the request to the correct location. (This would all be so much simpler if we could use custom headers when making a JSONP request in the first place. We'd need only the header and wouldn't have to screw with the URL at all.)

sub vcl_recv { 
        if (req.url ~ "^/jsonp/get/") { 
                set req.request = "GET"; 
                set req.http.x-rest-method = "JSONP"; 
                set req.url = regsub(req.url, "^/jsonp/get", ""); 
        } 
        if (req.url ~ "^/jsonp/put/") { 
                set req.request = "PUT"; 
                set req.http.x-rest-method = "JSONP"; 
                set req.url = regsub(req.url, "^/jsonp/put", ""); 
        } 
        if (req.url ~ "^/jsonp/post/") { 
                set req.request = "POST"; 
                set req.http.x-rest-method = "JSONP"; 
                set req.url = regsub(req.url, "^/jsonp/post", ""); 
        } 
        if (req.url ~ "^/jsonp/delete/") { 
                set req.request = "DELETE"; 
                set req.http.x-rest-method = "JSONP"; 
                set req.url = regsub(req.url, "^/jsonp/delete", ""); 
        } 
} 

So now the server will see the request just as it would from any other client, blissfully unaware that it had unclean origins. Well, there's still the parameters-as-GET-querystring issue. That may be problematic for some. If your server is written using software that allows you not to care where the parameters came from, e.g.
Catalyst::Request->params, or $_REQUEST in php, you're golden.

So now for (2):

sub vcl_deliver { 
        if (req.http.x-rest-method ~ "JSONP") { 
                set resp.status = 200; 
        } 
} 

If the request originated from JSONP, we fix the HTTP status of the response to 200, whatever it was. This implies that you're providing the client some sort of indicator of error in the body. But shouldn't you be anyway?

Of course, with varnish, you could just make the REST API look same-origin and obviate JSONP:

backend default { 
  .host = "page.mysite.com"; 
  .port = "80"; 
} 
 
 
 
backend myapi { 
  .host = "myapi.services.mysite.com"; 
  .port = "80"; 
} 
 
 
 
sub vcl_recv { 
        if (req.url ~ "^/myapi/") { 
                set req.backend = myapi; 
                set req.url = regsub(req.url, "^/myapi", ""); 
        } 
} 
 
In this example, I've prepended the name of the API to the URL. It appears to the requester as page.mysite.com/myapi, but is routed to myapi.services.mysite.com/ by varnish.

If these proxy layer solutions are not an option, try something like easyXDM. It will probably make your Javascript cleaner, and still protects the integrity of the service-layer code.