I would like to thank everyone who provided feedback on this grant, both on our blog and during the consultation phase.

Before we vote on this proposal we would like to get feedback and endorsements from the Perl community. Please leave feedback in the comments or send email with your comments to karen at perlfoundation.org.

Name: Jess Robinson

Project Title: Improving Cross compilation of Perl 5

Synopsis:

Perl 5 has several scripts and tools for cross-compiling it for other platforms. These are somewhat inconsistent and scattered across the codebase. My aim is to complete a cross-compilation target for Android, with smoker, and to prepare and document ways to add more targets without specialising for each of them.

The implementation goal is to allow direct cross compilation (no changes to actual Perl C-code, just using the hints system and configuration of cross-compiler paths and flags) to as many current and future platforms as possible, without needing knowledge of the build system internals. Ideally this will automate as much as possible the collection of data for the config files. Only platforms on which we can't run arbitrary compiled C code (eg Android) should get pre-defined config files in core, and if these become unmaintained (not updated when new features are added to Configure) they should be flagged as stale.

Benefits to Perl 5 Core Maintenance:

This work will remove duplicate build scripts and "specialised" code created for various cross compilation environments and either them replace with a generic version, or leave them out completely. Either way this will allow Core contributors to have a better overview of which specialised code needs to be maintained.

Deliverable Elements:

I propose to work this as a pay-as-you-go grant, similar to Nicholas and Dave. The initial request is for 200 hours at $50/hour, over 2 months. If the project proves to need less time than this, we may consider it done and end early, with the payment covering the actual number of hours worked. Alternately, if it requires more time I would be willing to work longer at the same rate to complete unfinished tasks.

The actual deliverables will be determined by the project, but will cover at least the delivery of a working Android cross-compilation build environment, together with a smoke environment to ensure that it continues to compile.

I will deliver weekly summary reports on progress via the p5p mailing list, and an end of month report summarising the whole month. I would then claim for the hours worked in that month.

The grant managers would be allowed at any time to inform the board that the project is failing, and the TPF board, after allowing me to present my side, decide to terminate the project.

Project Details:

I intend to spend time on the following items:

1. Bring my existing Android cross compiling environment and copy of Perl up to date and working. The current branch is based on perl 5.15.1, records work in progress, and uses an NDK two versions out of date. The branch needs rebasing and some restructuring to eliminate extraneous intermediate stages. The version of NDK used doesn't upgrade cleanly to newer ones. Fixing this will require trying to make the use of the NDK more generic and easier to configure in case of more path/binary renaming in future versions. approx 50 hours.

2. Set up a smoker for Android cross-compilation to ensure that the build continues to work. - 10 hours.

3. Refactor the working Android cross compiling approach to minimise divergence from the regular hosted build process, simplifying it as much as possible.

In particular:

• fully exploit the fact that it now uses ./Configure -Dmksymlinks to separate the host and target build trees, which wasn't done previously.
• use lib/buildcustomize.pl to simplify the cases where the host miniperl driving the build needs to report configuration values for the target system.
• assume that a functional /usr/bin/perl is present when cross-compiling, and convert top level driver scripts from /bin/bash to Perl for easier maintenance and improved portability.

[Flexible guess. - 30 hours]

4. Identify a second target platform to cross compile to, representative of a small consumer device running Linux or *BSD on a non-x86 chip, and which can be run under emulation for testing. One plausible candidate is the Raspberry Pi, an ARM Linux device, because developer virtual machines exist, are documented and actively maintained. Get cross compilation to this device working, using the toolchain documented in Cross/README.new, and also using the approach documented in INSTALL. - Unknown.

Refactor the cross compilation code described in the previous steps to

• take advantage of the simplifications possible with -Dmksymlinks and buildcustomize.pl approaches used by the Android port
• minimise the amount of duplication and overlap with the regular hosted build process
• share with the Android port as much code as is practical
• maximise maintainability
• aim toward an approach that where possible is generic, minimising the use of pre-requisite "canned" config files.

5. Document this with a "Howto cross-compile Perl" guide.

6. Set up a smoker for cross-compilation to the small-unix target to ensure that the build continues to work. - 10 hours.

7. Review the various Cross compilation scripts and tools existing today in the core code, to determine which are still required using the revised approaches above. Tidy up by removing / deprecating un-used or obsolete parts.

Stretch goals:

1. Devise and implement a way to perform an installation of the resulting build onto the target smoke platforms (using an emulator), run the perl 5 core test suite, and gather the results to report back to the current smoke system. - Unknown

2. Investigate ways to cross-compile CPAN modules, especially XS modules, and install onto the target device. Likely this would be several steps, starting with modules that don't probe the local system within their Makefile.PL, and build them as part of the main cross build. Then work to decouple building modules from building the perl core, possibly including making the cross-building setup installable on the host machine. - Unknown, estimate 25 hours.

I estimate that the main items 1 to 7 will take around 200 hours to complete. I will work on each item in turn (though not necessarily in the above order) and in stages to provide usable patches during the project.

These items were prepared with much help from Nicholas Clark, many thanks to him.

Project Schedule:

I expect to be working on this alongside other work, and realistically to be spending about half my time on it. Thus I am asking for 200 hours over 2 months.

I am available to start work on this immediately.

Bio.:

I have worked with Perl professionally for the last 7 years, and I spend much of my spare time supporting and encouraging the use of Perl.

I took the Perl 5.10 changes and additions from the android-scripting repo and ported them to enable cross-compiling of Perl 5.15 for Android devices last year, this work can be seen on github: https://github.com/castaway/perl/tree/x-compile-android.

Endorsed by: Nicholas Clark, Ricardo Signes

Amount Requested: $10,000

A short report this month as less work got done, due to illness and disruption.

The single largest item of the month was spent getting my head around the state of cross compilation. Mostly this was figuring out the history and state of various cross compilation approaches in the core codebase, to help Jess Robinson refine a potential TPF grant application for cross compilation (particularly to Android). At the time of writing most of this is still groundwork (and intangible), such as studying her x-compile-android work at https://github.com/castaway/perl and the various existing *nix cross-compilation approaches in the core, but hopefully it should soon materialise into the actual published grant application, approval, and then visible code and progress. We think we can see an approach to take that has a good chance of working out long term, by being as simple as possible within the existing build setup.

I also diagnosed the likely cause of File::Glob failures we're seeing on some HP-UX machines. It looks like another C compiler bug handling bool. Pesky new fangled C99 and its complex new features. 12 years is clearly not long enough for compiler writers to be sure to have all the bugs squished. However, I've not had time to investigate further and be confident that my initial "top of my head" fix is the best long term plan, so this build failure is not yet resolved.

I spent some time starting to prune the core's todo list. Historically, the core's list of todos was recorded in the perltodo man page. This really isn't the best place for it, as installing snapshot that becomes increasingly out of date isn't that useful to anyone. So the 5.15.9 is the end of an era in this respect. The todo list is now in Porting/todo.pod, and the man page replaced with a small placeholder that points the reader to the git repository for the current version. Currently the version in blead is unchanged, but in a branch I've culled about 10% of the todos, those that are either done, mostly done, or no longer relevant. There are still a few to be updated where we have a better idea of how to do it, and the document needs restructuring to make it easier to spot appropriate "starter" tasks.

Whilst looking at some older bugs I needed to build various CPAN modules on OpenBSD against older versions of perl. Bootstrapping (older) CPAN didn't turn out be as easy it could have been, partly thanks to command line tools somehow managing to double gzip tarballs. The following approach might be useful to others in a similar situation on a recalcitrant setup:

1) manually get a tarball of the current CPAN.pm onto the machine and unpack it

2) Run its Makefile.PL. For now, don't worry about the missing pre-requisites warnings. Build it. (Run tests.) Install it.

3) Run the newly installed cpan client on the checked out tarball, as 'cpan .' If you need proxies for HTTP and FTP, set HTTP_PROXY and FTP_PROXY appropriately in your environment. Configure the CPAN client as prompted. This will fetch the missing pre-requisites, build and install them.

This approach is far more likely to succeed than trying to get the older version's installed cpan client to get a new CPAN.pm, as the older version doesn't incorporate the bug fixes and work arounds for awkward command line tools.

A more detailed breakdown summarised from the weekly reports. In these:

16 hex digits refer to commits in http://perl5.git.perl.org/perl.git
RT #... is a bug in https://rt.perl.org/rt3/
CPAN #... is a bug in https://rt.cpan.org/Public/
BBC is "bleadperl breaks CPAN" - Andreas König's test reports for CPAN modules
ID YYYYMMDD.### is an bug number in the old bug system. The RT # is given afterwards.*

73.00 hours total
* One can convert old system numbers to RT ticket numbers using the form at
https://rt.perl.org/perlbug/ The old system numbers are mostly a historical curiosity, but they can be useful when searching git logs and the mailing list archives.

Sorry for the delay in this report - I've been ill, been away, been ill while away, and generally disrupted, and I wasn't the only one ill in the house. Fortunately everyone is well again, and the backlog of other tasks is considerably reduced.

Zombie undead global backref destruction panics strike again. I don't know what it is with these critters, but another variant turned up, in almost the same place, and triggered by almost the same code. I think that this is the last one, simply because this was the third and final branch of the backref code, and now it's been diagnosed and fixed.

In this case, the problem is that it's possible for the the last (strong) reference to the target [tsv in the code in Perl_sv_del_backref()] to have become freed before the last thing holding a weak reference. If both survive longer than the backreferences array (the previous cause of problems), then when the referent's reference count drops to 0 and it is freed, it's not able to chase the backreferences, so those backreferences aren't NULLed.

For example, a CV holds a weak reference to its stash. If both the CV and the stash survive longer than the backreferences array, and the CV gets picked for the SvBREAK treatment first, and it turns out that the stash is only being kept alive because of an our variable in the pad of the CV, then midway during CV destruction the stash gets freed, but CvSTASH isn't set to NULL. It ends up pointing to the freed HV. Hence that pointer is chased into Perl_sv_del_backref(), but because it's pointing to a freed HV the relevant magic structure was no longer there to be found, a NULL pointer was assigned to a local variable. Subsequent code panicked because it thought that could never happen, at least not without a bug. Except, as the investigation showed, it could happen quite legitimately in exactly this scenario. During global destruction, all bets are off.

I don't believe that "better" destruction ordering is going to help here - during global destruction there's always going to be the chance that something goes out of order. We've tried to make it foolproof before, and it only resulted in evolutionary pressure on fools. Which made us look foolish for our hubris. :-(

I think that the reason that all these critters are shuffling towards us*now*, despite being in code that's quite long lived, is because since 5.14.0 Dave has re-worked the SV destruction code. Previously it would recurse into data structures, which had the unpleasant side effect of blowing the C stack when it tried to do too much at once. [Crash and burn - not good] Dave has made much of that code iterative now, which avoids the crashing. [Generally this is seen as progress :-)] However, it's changed the destruction order, and I think in some cases that is exposing long-latent bugs elsewhere in the code that destruction calls, particularly during global destruction.

Another signficant activity in February week was diagnosing and resolving ticket #37033. The low bug number reveals that this is quite a long standing bug, related to the parser leaving file handles open in some cases. The background is that the perl interpreter is either passed a filename, or if none is passed defaults to reading from STDIN. If reading from STDIN, the interpreter should not close it, as it did not open it (and implicitly closing STDIN out from underneath a program is bad). But any file handle the interpreter opened (internally, as a side effect of doing its job) should be closed.

The problem was the the check for close-or-not used to be simply "is this filehandle STDIN?"

The troubles arise from the interaction of the definition of STDIN with the POSIX semantics of open. "is this STDIN?" is pretty much "is this file handle open on file descriptor 0?". open gives you the lowest unused file descriptor. So if the user program closes STDIN, then the next file handle that is opened meets the "is this STDIN?" test. And the parser was mistaking this for the situation where it had defaulted to reading the program from STDIN because no filename was passed in, so it was leaving the file handle open. Of course, this is a leak. However, it was also starting to show up as visibly buggy behaviour in code that manipulated STDIN - close STDIN, do some "innocent" calculation, (re)open STDIN, only it's not on file descriptor 0 now - what's wrong?

What changed? Karl's work in improving Unicode semantics means that several ops now need look up certain Unicode properties in some cases. If these aren't cached yet, there's a call back into the parser as part of the loading routines. And if that call into the parser happened while STDIN was closed, oh dear, come close time the parser used to thing that had been defaulting to reading from STDIN, and left the handle open.

The parser now explicitly tracks what it opened, and hence closes everything correctly.

A side effect of tracking all this down was that I started to look at the code in perl.c that handles the parsing of the command line options and the initialisation they trigger, along with the the handling of the "script file name", the "-e" option and the default to STDIN. Some of the cleanup has been committed to blead and will be in 5.16.0. Other parts are in the branches nicholas/RT37033-followup and smoke-me/kick-FAKE_BIT_BUCKET However it remains a work in progress - right now perl needs to open /dev/null as part of -e handling, because of assumptions in the parser. However, I think I can see roughly how to make a small change to the source filter code that would permit -e to avoid needing to open any file handle.

installhtml also forced it way onto my plate again in February. This was mysteriously passing its tests on George Greer's Win32 setup that smokes "smoke-me" branches, and yet failing one test on George Greer's Win32 setup that smokes blead. Same operating system, same code, same everything, so why isn't it the same result? After quite a lot of head scratching, code chasing (you are in a twisty maze of abstraction layers, all alike), it turns out that the problem is that on one VM, the smoker is configured to smoke at a pathname D:\path\to\smoker and on the other VM its configured to smoke at d:\path\to\smoker. That difference is almost irrelevant, because filenames on Win32 are* case preserving, case insensitive, and most things aren't concerned with which name a file has, merely whether they can open (or delete) it.

However, Pod::Html has an explicit white-box test for its caching. That test needs to be sure that the right pathnames are in the cache index. And it turns out that the pathnames written to the cache are canonicalised by File::Spec, whilst the expectation list built by the test is not. And that whilst files and directories on Win32 are case preserving, and hence unchanged on canonicalisation, drive letters are canonicalised to upper case. Yay.

I wonder what useful stuff gets displaced from my brain by this obscure knowledge? I'm sure I used to be able to remember the lyrics to American Pie... :-(

To finish the month, I finished a chunk of work related to a regression introduced by commit 6634bb9d0ed117be. A side effect of the improvements in that change was a small regression in how code such as this would deparse:

use 5.10.0;
say "Perl rules";

It should deparse as C - instead it started to be deparsed as C. Of course nothing in life (or at least in perl 5) is simple, as verifying that this is fixed turned out to be something the B::Deparse tests couldn't actually do - they were structured in such a way that they effectively ran with an implicit C at the top level. So first the B::Deparse test infrastructure had to be refactored to remove that, and tests that were relying on it fixed.

A more detailed breakdown summarised from the weekly reports. In these:

16 hex digits refer to commits in http://perl5.git.perl.org/perl.git
RT #... is a bug in https://rt.perl.org/rt3/
CPAN #... is a bug in https://rt.cpan.org/Public/
BBC is "bleadperl breaks CPAN" - Andreas König's test reports for CPAN modules

139.50 hours total

As per my grant conditions, here is a report for the March period.

Spent the month successfully bringing run-time regex code blocks into the new way of doing things.

My previous work on this had handled the cases where the code appears within a literal regex, e.g.

/ab(?{})/;
$r = qr/xy(?{})/;
/z$r/;

In all the above cases, the code block is parsed and compiled by the perl parser at the same time as the surrounding code is compiled; the compiled blocks are then retained and passed to the regex engine when the pattern is compiled, and preserved when for example a qr is interpolated within another pattern, as above.

My changes this month extend this to run-time patterns, e.g.

use re 'eval';
my $code = '(?{})'; /$code/;
$foo =~ '(?{})'; # this is a string literal, not a regex literal

It does this by making a copy of the pattern string, but with any literal code blocks blanked out and \ and ' escaped; it then calls eval_sv() on an SV containing the string
qr'pattern with \\,\' escapes and literal (?{}) blanked out' then any compiled code blocks are extracted from the qr and added to the pool of code blocks (e.g. from literal code blocks already compiled).

Having reached this point, it means I could then remove all the old-style re_eval compiling code in the core, including the hated sv_compile_2op() function, and considerably simplifying S_doeval().

There are three more things I still need to do: the main one is to fix the way code blocks are invoked, chiefly to give them a proper entry on the context stack so that things like die,next,and caller don't crash, but also to make recursion work properly. The other two things are to do a big list of miscellaneous fixes and tweaks that I've been noting as I go along; and finally, to check that all the tickets attached to the meta-ticket have in fact been fixed by these changes.

Over the last month I have averaged 16 hours per week.

As of 2012/03/31: since the beginning of the grant:

108.4 weeks
1139.9 total hours
10.5 average hours per week

There are now 160 hours left on the grant.

Report for period 2012/03/01 to 2012/03/31 inclusive

Summary

Effort (HH::MM):

0:00 diagnosing bugs
69:10 fixing bugs
0:00 reviewing other people's bug fixes
0:00 reviewing ticket histories
0:00 review the ticket queue (triage)
-----
69:10 Total

Numbers of tickets closed:

1 tickets closed that have been worked on
0 tickets closed related to bugs that have been fixed
0 tickets closed that were reviewed but not worked on (triage)
-----
1 Total

Short Detail

67:40 [perl #34161] METABUG - (?{...}) and (??{...}) regexp issues
1:30 [perl #111974] 5.15.9 breaks Glib