I noticed that CLISP (2.44.1) and sbcl (1.0.18.debian) differ in
detaills of the output of (READ).
E.g. for `(,a b)
CLISP's (READ) yields:  (BACKQUOTE (((UNQUOTE (A NIL)) (B NIL)) NIL))
while sbcl yields: (BACKQ-CONS (A ((QUOTE ((B NIL) NIL)) NIL)))

Is this because either or both implementations aren't standard
compliant or because the ANSI CL standard doesn't define the result of
READ (and consequently the input to EVAL) precisely? If the latter,
how can macros be implemented in a portable manner?

G�nther Thomsen wrote:
> I noticed that CLISP (2.44.1) and sbcl (1.0.18.debian) differ in
> detaills of the output of (READ).
> E.g. for `(,a b)
> CLISP's (READ) yields:  (BACKQUOTE (((UNQUOTE (A NIL)) (B NIL)) NIL))
> while sbcl yields: (BACKQ-CONS (A ((QUOTE ((B NIL) NIL)) NIL)))
> 
> Is this because either or both implementations aren't standard
> compliant or because the ANSI CL standard doesn't define the result of
> READ (and consequently the input to EVAL) precisely? If the latter,
> how can macros be implemented in a portable manner?

 From the Hyperspec, 2.4.6.1 [1]:  "Since the exact manner in which the 
Lisp reader will parse an expression involving the backquote reader 
macro is not specified, an implementation is free to choose any 
representation that preserves the semantics described."

Also see 2.4.6 [2]:  "An implementation is free to interpret a 
backquoted form F1 as any form F2 that, when evaluated, will produce a 
result that is the same under equal as the result implied by the above 
definition, provided that the side-effect behavior of the substitute 
form F2 is also consistent with the description given above. The 
constructed copy of the template might or might not share list structure 
with the template itself. As an example, the above definition implies that

`((,a b) ,c ,@d)

will be interpreted as if it were

  (append (list (append (list a) (list 'b) 'nil)) (list c) d 'nil)

but it could also be legitimately interpreted to mean any of the following:

  (append (list (append (list a) (list 'b))) (list c) d)
  (append (list (append (list a) '(b))) (list c) d)
  (list* (cons a '(b)) c d)
  (list* (cons a (list 'b)) c d)
  (append (list (cons a '(b))) (list c) d)
  (list* (cons a '(b)) c (copy-list d))"

What you can be sure of is that the results of evaluating

  (BACKQUOTE (((UNQUOTE (A NIL)) (B NIL)) NIL))

in CLISP and

  (BACKQ-CONS (A ((QUOTE ((B NIL) NIL)) NIL)))

in SBCL will be lists with EQUAL structure.

//JT

[1] http://www.lispworks.com/documentation/HyperSpec/Body/02_dfa.htm
[2] http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm

 GT> Is this because either or both implementations aren't standard
 GT> compliant or because the ANSI CL standard doesn't define the result of
 GT> READ (and consequently the input to EVAL) precisely?

Are you sure that you understand what backquote is? It is a convenience
 read macro which helps constructing forms like (list a (quote b)).

Standard defines what READ does -- once it encounters backquote,
it will call a read macro associated with it. That would be either
user-defined reader macro or standard reader macro.

Details how exactly backquote reader macro works are not of a great
importance -- it can do anything as long as it works and satisfies the
spec. So if it differs it is not a problem at all.

By the way, it looks like you do not really understand how it all works.
Standard does not define results of READ at all -- it defines what
READ is supposed to do. Results of READ depend on a lot of things, like
a current readtable etc. And you can construct a form which is
read in different way each time, e.g.

#.(random 1.0)
'#.(gensym)

Obviously, you cannot preciously define what _result_  should be for such 
forms,
yet it is fully specified how does it work.

 GT>  If the latter, how can macros be implemented in a portable manner?

I barely can imagine macros which depend on how backquote is read.
Would you provide an example? 

On Jan 17, 4:13=A0pm, Joshua Taylor <tay...@cs.rpi.edu> wrote:
[..]
>
> What you can be sure of is that the results of evaluating
>
> =A0 (BACKQUOTE (((UNQUOTE (A NIL)) (B NIL)) NIL))
>
> in CLISP and
>
> =A0 (BACKQ-CONS (A ((QUOTE ((B NIL) NIL)) NIL)))
>
> in SBCL will be lists with EQUAL structure.
>
Ah, I think, that's the key, I was missing. Thanks.

On Jan 17, 4:13=A0pm, Joshua Taylor <tay...@cs.rpi.edu> wrote:
[..]
>
> What you can be sure of is that the results of evaluating
>
> =A0 (BACKQUOTE (((UNQUOTE (A NIL)) (B NIL)) NIL))
>
> in CLISP and
>
> =A0 (BACKQ-CONS (A ((QUOTE ((B NIL) NIL)) NIL)))
>
> in SBCL will be lists with EQUAL structure.
>
> //JT
>
Yes, the results of the evaluation better be the same. If I understand
macro evaluation however correctly (and no, I'm not sure about that)
they are applied during the evaluation process, i.e. they will work
with the input to EVAL (the output from READ). Even though I cannot
readily come up with a macro depending on the presence of the
backquote/back-cons/back-list form, I sense some danger there. I also
stumbled just accidentally upon this difference and wonder, if there
are other cases, perhaps some which more likely will interfere with
practical macros.

G�nther Thomsen <guenthert@gmail.com> writes:

> On Jan 17, 4:13�pm, Joshua Taylor <tay...@cs.rpi.edu> wrote:
> [..]
>>
>> What you can be sure of is that the results of evaluating
>>
>> � (BACKQUOTE (((UNQUOTE (A NIL)) (B NIL)) NIL))
>>
>> in CLISP and
>>
>> � (BACKQ-CONS (A ((QUOTE ((B NIL) NIL)) NIL)))
>>
>> in SBCL will be lists with EQUAL structure.
>>
>> //JT
>>
> Yes, the results of the evaluation better be the same. If I understand
> macro evaluation however correctly (and no, I'm not sure about that)
> they are applied during the evaluation process, i.e. they will work
> with the input to EVAL (the output from READ). Even though I cannot
> readily come up with a macro depending on the presence of the
> backquote/back-cons/back-list form, I sense some danger there. I also
> stumbled just accidentally upon this difference and wonder, if there
> are other cases, perhaps some which more likely will interfere with
> practical macros.

About your question whether macros can be implemented in a conformant
way, obviously to do so, your macro shouldn't depend on a specific
result from (car '`(,(+ 1 2))).

Usually, backquote is used to build the result of the macro, and the
result of the macro will normally be used by CL:EVAL, and the
implementation presumably it will be able to handle conformably its
own backquote.

However, you could indeed use macroexpand or macroexpand-1, or try to
process the result of a backquote form in macros (or other functions).
In order to do that conformably, you would have to respect the rules
enounced in section "1.5.2 Conforming Programs".

Notably, since the result of a backquote form is a form, you would
have to treat it as any other form, eg. using it to build a bigger
form, code-walking it, or evaluating it (or rather, return it from a
macro so that the compiler can compile it).


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

G�nther Thomsen  <guenthert@gmail.com> wrote:
+---------------
| Joshua Taylor <tay...@cs.rpi.edu> wrote:
| > What you can be sure of is that the results of evaluating
| >   (BACKQUOTE (((UNQUOTE (A NIL)) (B NIL)) NIL))
| > in CLISP and
| >   (BACKQ-CONS (A ((QUOTE ((B NIL) NIL)) NIL)))
| > in SBCL will be lists with EQUAL structure.
|
| Yes, the results of the evaluation better be the same. If I understand
| macro evaluation however correctly (and no, I'm not sure about that)
| they are applied during the evaluation process, i.e. they will work
| with the input to EVAL (the output from READ). Even though I cannot
| readily come up with a macro depending on the presence of the
| backquote/back-cons/back-list form, I sense some danger there. I also
| stumbled just accidentally upon this difference and wonder, if there
| are other cases, perhaps some which more likely will interfere with
| practical macros.
+---------------

It almost never interferes with CL macros the way that CL users usually
write macros. So in that sense, it's a "Don' worry 'bout it!". However...

There are two scenarios in which your macro -- the one you wrote in
CL that somebody else[1] is calling -- might need to know the exact
details of what READ returned from a backquoted [a.k.a. quasiquoted]
form, namely:

1. When your macro is itself doing parsing/rewriting of its parameters
   [especially the &BODY parameter] either to achieve some low-level
   optimization or to implement some DSL. In such cases, your macro
   might end up having to do [or calling some implementation-dependent
   auxiliary function to do] a full code walk of the actual parameter(s),
   with all the attendent hair that implies.

   Doing such code walking during macro expansion has been discussed
   here many times, with at least one long thread fairly recently.

2. When your macro is trying to use only *part* of the backquote
   [a.k.a. quasiquote] syntax for its own purposes [usually a DSL].
   The example of most interest to me personally is the various
   attempts[2] to implement SCSH [the Scheme Shell] in Common Lisp.
   You simply *can't* -- at least not on most CL implementations and
   certainly not portably -- but if you tweak the SCSH input syntax
   in a tiny but critical way [that makes it *incompatible* with
   original SCSH code, note!] you can get very close.

To say it a different way:

a) If all of the backquotes that are written are in the *definition*
   of your macro(s), then "don' worry 'bout it!"  Your implementation's
   READ & EVAL or READ & COMPILE will work together correctly, no matter
   what READ returns from a backquoted form.

b) If some of backquotes that are written are in the *use* of your
   macro(s) [especially the actual &BODY parameter], you're still
   probably fine... unless your macro starts digging into the guts
   of the parameter(s) [instead of just substituting them], in which
   case you're on your own!!  :-{


-Rob

[1] The "somebody else" includes *you* at some later date, of course! ;-}

[2] Including mine! ;-}  ;-} See this thread:

      http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/98d32d48786d5670/00545caa0de7bf0b#00545caa0de7bf0b

    Or if that doesn't work for you, this individual message within it
    describes the SCSH-porting issue (especially see footnotes #2 & #3):

      http://groups.google.com/group/comp.lang.lisp/msg/00545caa0de7bf0b

-----
Rob Warnock			<rpw3@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607

rpw3@rpw3.org (Rob Warnock) writes:
> 2. When your macro is trying to use only *part* of the backquote
>    [a.k.a. quasiquote] syntax for its own purposes [usually a DSL].
>    The example of most interest to me personally is the various
>    attempts[2] to implement SCSH [the Scheme Shell] in Common Lisp.
>    You simply *can't* -- at least not on most CL implementations and
>    certainly not portably -- but if you tweak the SCSH input syntax
>    in a tiny but critical way [that makes it *incompatible* with
>    original SCSH code, note!] you can get very close.

In this case, nothing prevents you to write a different reader macro
for #\,.

> b) If some of backquotes that are written are in the *use* of your
>    macro(s) [especially the actual &BODY parameter], you're still
>    probably fine... unless your macro starts digging into the guts
>    of the parameter(s) [instead of just substituting them], in which
>    case you're on your own!!  :-{

Actually, you do not have to do anything more, or anything less, to
process a backquote form than to process any other body form.

If you are already "digging into the guts" of the body parameter
correctly, then you are already processing backquotes correctly:

(defmacro gut-digging (&body body)
   `(progn ,@(mapcar (function dig-into-the-guts) body)))

(let ((here 42)
      (there '(1 2 3)))
  (gut-digging `(ha ha! ,here and ,@there)))


The reason why is because CL implementations cannot provide new
special operators without providing an equivalent macro definition, so
that code walking is possible.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

Pascal J. Bourguignon <pjb@informatimago.com> wrote:
+---------------
| rpw3@rpw3.org (Rob Warnock) writes:
| > 2. When your macro is trying to use only *part* of the backquote
| >    [a.k.a. quasiquote] syntax for its own purposes [usually a DSL].
| >    The example of most interest to me personally is the various
| >    attempts[2] to implement SCSH [the Scheme Shell] in Common Lisp.
| >    You simply *can't* -- at least not on most CL implementations and
| >    certainly not portably -- but if you tweak the SCSH input syntax
| >    in a tiny but critical way [that makes it *incompatible* with
| >    original SCSH code, note!] you can get very close.
| 
| In this case, nothing prevents you to write a different reader macro
| for #\,.
+---------------

Well, yes, of course. But in that case you have to re-implement *all*
of the functionality of #\, and the others yourself. Not fun.

The key to simple processing of the SCSH syntax was that in Scheme
the reader *only* rewrites `FORM into (QUASIQUOTE FORM), leaving the
rest of the processing to EVAL and the QUASIQUOTE syntax [or macro].
This means that one can write ordinary macros that provide what might
be called "implicit quasiquoting" [as many of the SCSH syntactic forms
require]; such macros can simply wrap a (QUASIQUOTE ...) around one or
more of their argument (sub)forms and let macroexpansion and/or EVAL
do the rest.

Whereas in CL, a ,FORM or ,@FORM not properly contained within an
outer `FORM is illegal, so [as you say] you must provide your *own*
complete readmacro re-implementation of ,FORM and ,@FORM *and* `FORM
if you want to allow macros that provide "implicit quasiquoting".
Plus, once you do that you lose the ability to access whatever
optimizations that the implementation's builtin backquote processing
might provide.

+---------------
| > b) If some of backquotes that are written are in the *use* of your
| >    macro(s) [especially the actual &BODY parameter], you're still
| >    probably fine... unless your macro starts digging into the guts
| >    of the parameter(s) [instead of just substituting them], in which
| >    case you're on your own!!  :-{
| 
| Actually, you do not have to do anything more, or anything less,
| to process a backquote form than to process any other body form.
+---------------

Oh, really?!? You are *much* more likely to run into strange
implementation-dependedent macros and/or functions when processing
a backquote form than with other forms. [See below.]

+---------------
| If you are already "digging into the guts" of the body parameter
| correctly, then you are already processing backquotes correctly:
| 
| (defmacro gut-digging (&body body)
|    `(progn ,@(mapcar (function dig-into-the-guts) body)))
| 
| (let ((here 42)
|       (there '(1 2 3)))
|   (gut-digging `(ha ha! ,here and ,@there)))
| 
| The reason why is because CL implementations cannot provide new
| special operators without providing an equivalent macro definition,
| so that code walking is possible.
+---------------

But CL implementations are free to implement backquote forms with
non-standard functions & compiler macros, even *without* introducing
new special operators, e.g., in CMUCL[1]:

    cmu> (car (read-from-string "`(ha ha! ,here and ,@there)"))

    LISP::BACKQ-LIST*
    cmu> (describe 'lisp::backq-list*)

    BACKQ-LIST* is an internal symbol in the LISP package.
    Function: #<Function LISP::BACKQ-LIST* {10442F91}>
    Function arguments:
      (&rest args)
    ...
    Documentation on the compiler-macro:
    NIL
    cmu> (compiler-macro-function 'lisp::backq-list*)

    #<Function (:COMPILER-MACRO LISP::BACKQ-LIST*) {104430D9}>
    cmu> 

Whereas in the case of CLISP you get a plain macro[2], not a function
[and no compiler macro either]:

    [1]> (car (read-from-string "`(ha ha! ,here and ,@there)"))
    SYSTEM::BACKQUOTE
    [2]> (describe 'system::backquote)

    SYSTEM::BACKQUOTE is the symbol SYSTEM::BACKQUOTE, lies in
    #<PACKAGE SYSTEM>, is accessible in the package SYSTEM, names a macro.
    ...
    [3]> (compiler-macro-function 'system::backquote)
    NIL
    [4]> 

So clearly your GUT-DIGGING function is going to have to have
implementation-dependent cases for all of variations of backquote
across all of the implementations for which you want it to be portable.


-Rob

[1] The full return value from READ in CMUCL:

    (LISP::BACKQ-LIST* (QUOTE HA) (QUOTE HA!) HERE (QUOTE AND) THERE)

[1] The full return value from READ in CLISP:

    (SYSTEM::BACKQUOTE (HA HA! (SYSTEM::UNQUOTE HERE) AND
			(SYSTEM::SPLICE THERE)))

-----
Rob Warnock			<rpw3@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607

rpw3@rpw3.org (Rob Warnock) writes:

> Oh, really?!? You are *much* more likely to run into strange
> implementation-dependedent macros and/or functions when processing
> a backquote form than with other forms. [See below.]

Yes, but they'll be code-walkable.


> So clearly your GUT-DIGGING function is going to have to have
> implementation-dependent cases for all of variations of backquote
> across all of the implementations for which you want it to be portable.

Well, my point is that gut-digging cannot and must not depend on the
actual structure of the code it's given to dig gut.  All it can do, is
code-walk, and identify subforms which it may want to substitute,
and any code transformation it would do to other function calls or
special operators (having expanded all the macros), would be done the
same for those returned by backquote as those provided otherwise by
the programmer.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

You might want to look at this.
http://paste.lisp.org/display/93608
Load it into your implementation, set up macro characters
and be sure bacquotes are processed in an implementation-independent
manner.
Feedback would be appreciated.

> Feedback would be appreciated.
Note that '`(foo ,bar) won't be printed
as `(foo ,bar). This should be fixed.
To do so, we would need to encode quasiquoting
syntax elements by defstrcuts (one defstruct per
syntax element) and add print-functions to them.
I think this can be fixed easily, but I have no time
for that now.