Inline.pm Notes

Perl 's Inline.pm (Inline::C, etc), and Python 's PyInline , permit easy on-demand compilation and linking of C code. As of 1st Quarter 2002, many folks still think of them as merely a way to generate glue code, Inline.pm's original motivation, rather than as the first widely available run-time code generation system. And thus as a powerful tool with deep implications for the speed and architecture of scripting language software. This page is part of my efforts to change that perception. Hopefully it will be obsolete by 2003. Comments are encouraged .

Links

(Inline as part of a "sea change" in multilingual programming is a topic for another page.)

Some local pages
An Illustration of Perl Objects with C APIs
An Optimization Anecdote about PyInline, a python-list post.
Perl Objects with C API's - or writing C extensions in an Inline world Rough slides from an informal talk.
Using Inline::CPP with wxPerl post.

A quick example

sub set_element {
  my($self,$n,$value) = @_;
  substr($self,$n,1) = pack("C",$value);
}

def set_element(self,n,value):
  self.buffer[n] = struct.pack("B",value)

for(my $i=0;$i<1_000_000;$i++) {
  $object->set_element($i,67);
}

for i in range(1000000):
  object.set_element(i,76)

sub get_C_API {
  return "
#define USING(object) uchar *ptr = SvPVX(SvRV(object))
#define SET_ELEMENT(IDX,VAL) ptr[IDX] = VAL
";
}

def get_C_API(self):
  return """
#define USING(object) uchar *ptr; \
  object.ob_type->tp_as_buffer->bf_getwritebuffer(object,0,&ptr)
#define SET_ELEMENT(IDX,VAL) ptr[IDX] = VAL
"""

my $code = $object->get_C_API() . "
  void do_example (SV* object) {
    USING(object);
    int i;
    for(i=0;i<1000000;i++) { SET_ELEMENT(i,67); }
  }
";
Inline->bind(C => $code);
&do_example($object);

m = C.Builder(code = object.get_C_API() + """
  void do_example (PyBufferObject *object) {
    int i;
    USING(object);
    for(i=0;i<1000000;i++) { SET_ELEMENT(i,67); }
  }
""").build()
m.do_example(object.buffer)

And even better, because it is fast, we are not forced to clutter up the object api with a method set_all_elements. Nor with every other derivative method which someone, somewhere, might wish to run fast. These can be properly placed elsewhere (using mix-in classes, for instance), without performance penalty, and operate on any object which supports the particular C API (and using adapters, also on those which don't!).

Blazing speed and architectural clarity. Happiness.

Here is a similar example .

A larger example

The proposition

The current inflexibility of border placement has a pathological consequences. Because calling across the border is expensive, anything which must run fast, must join the C object. This "osmotic pressure" results in C objects with bloated APIs. An then in complexity like "cross-border request mini-languages". Further, the choice of problem decomposition, of object selection, is overridingly driven by performance issues, rather than productivity ones like simplicity, clarity, and ease of reuse and recombination.

So we have, for example, N different, incompatible, image libraries (2N with Python and Perl:). Each with their own 3 dimensional array of integers, their own embedded font library and drawing functions, and their own assorted _big_ apis. How much nicer it would be to have separate 3-spaces, font classes, and drawing classes. All naturally integrated with the normal object system. All orthogonal. All free to compete and evolve to well serve their separate domains. The 3-space with a simple auxiliary C API. Which classes and end-users could straightforwardly use to run just as fast as, and indeed faster than, with the current approach.

Software overview

While using Perl's Inline with a complex preexisting code base is currently ambitious, especially in languages other than C, it does currently seem plausible to make some use of Inline::C in production code.

Using perl's Inline from Python/TCL/etc

Notes

Dynamic code generation

Selective parsing

Blended language objects

Inheriting from Inline.pm

Why are these checks performed? Class Inline is currently playing two conflicting roles -- that of user interface, and that of base class for the assorted language modules (Inline::C, Inline::Java, etc, are all @ISA = 'Inline'). Little use is made of it in this latter role, but until it gets divested (into an "Inline::Language"? "Inline::ILSM"?), Inline croaks to control its UI. Specifically, to avoid being invoked via the language modules (eg, as Inline::C->bind() rather than Inline->bind(C=>...)).

One possible workaround is

  package Inline;

+ sub _class_is_equivalent_to_Inline { 0 }

-     if ($class ne 'Inline') {
+     if ($class ne 'Inline' and !$class->_class_is_equivalent_to_Inline()) {

-   croak M03_usage_bind() unless $class eq 'Inline';
+   croak M03_usage_bind() unless $class eq 'Inline'
+     or $class->_class_is_equivalent_to_Inline();

-      ref($o) eq 'Inline'
+      (ref($o) eq 'Inline' or $o->_class_is_equivalent_to_Inline())

  package InlineX;
  @ISA = qw(Inline);
  sub _class_is_equivalent_to_Inline { 1 }

Some assorted bitesized notes

Someone described this page as interprocedural optimization meets just in time compilation. But where the jit is both over run time, and over _development_ time.

Re bloated libraries - C code for interacting objects is far better combined at installation, compile, and run time, rather than at development time.

Who cares? For the speed obsessed, one gets _faster_ than normal C code via interprocedural optimization. For C coders in general, designing interfaces gets is easier, as they don't have to combine ease of use with power. Naive clients are off using derivative scripting-level interfaces, so the core api can focus on power. And because the C code is ephemeral, with maintainability coming from the scripting language level, one can have task specific apis, and can play grody code generation games one could never dream of when the C code has to be maintained directly. For example - "Want to completely change your api? No problem. Generating two is no big deal, and clients naturally ask for the version number of the api they wish to be handed". For scripting- language- only folks, who don't directly care about either speed or C, hopefully this will all enable other people to give you nicer objects.

Notes:
  I hope this page will be obsolete and stagnant by 2003.
  The page title may be unusually unstable.

Doables:
  api kiss illustration.  toy ppm, array fold, draw line.
  perl Inline from python
  fill in links.
  pull array lnks to new array section?  array apis.  lots of.
  notes on  memory allocation, 
  Inline inheriting post
  Inline::CPP parse filtering post, section
  demo mixin rtcg, do the architecture impact argument
    eg (seq props, transform/subspace)
  example gui image patches (Py/PlTcl, PyLatestGreatest, Gtk?)
  patterns of calling, ns, unlink, linker hell
  example rtcg'ed function, generic, class v instance apis.
  blended objects.  upcalls, C/XS/Perl, C++/Perl, exceptions.
    doubleended objects. multistep upcalls(C-Perl-Python).
  PNM `stroll through design space' story, generic font/graphics example
  metaprogramming, refactoring,
   need for Code - examples Inline LIB, C decl/cmds, hash combination
  examples mixin programming, PPMImage class,
  "The API Project" vis rtcg and P6 syntax flexibility.
     consequences of flexibility post
  rtcg, contrast current scripting w classical
  www.oonumerics.org/blitz/
  comparison and integration with C++ template games
  example use with prototype oo.
    and w Classless/MultiMethods/SelfMethods/Prototyped, MakeMethods?
  wavefront argument - Inline.pm as symptom and one of many crnt/historical, lispm
  timeframe box - jit, compile.  is engineering.
  single community argument, traditional dissipation, metaprog, apis
  time/mention/patch py's packed array class
  packed array implementations, pl/py.  mixins.
  iterators, array streams, subspaces
  inline vs weave design space
  sketch big picture?
  Psyco? Py2Cmod?

History:
 2002-Dec-19  Added links to CriTcl.
 2002-Oct-06  Added link to new Inline Wiki.
 2002-Oct-01  Added links to RubyInline, Ruby, perl man's Internals.
 2002-Sep-30  Added `Inline::SWIG'.  HTMLified `PyInline anecdote'.
 2002-Sep-23  Added `Using Inline::CPP with wxPerl' post.
 2002-Apr-13  Added pyperl download, documentation links.
 2002-Apr-09  Added `An Illustration of Perl Objects with C APIs'.  Ruby C api.
 2002-Apr-04  Added links to CGenerator, Pyrex.
 2002-Mar-13  Added slides from this evening's talk.  And link to Anecdote post.
 2002-Mar-09  Fixed link.  Minor edits.
 2002-Mar-06  Added "The proposition", "Some assorted bitesized notes".
 2002-Mar-05  Online.  "Links", "A simple example", "Inheriting from Inline.pm".