Below is my bare-bones sketch of an interface for modular programming
in Forth, adapted from the code of Bruce McFarling, David N. Williams,
and Neal Bridges. I think it provides the essential features necessary
for modular programming, with the exceptions of a wordlist to collect
the module references, and Bruce's HELPER words. The code is very
preliminary, and my testing has been only cursory, using the test
examples given below. Feel free to offer criticisms and suggestions.

Krishna
--
\  First, the demo/test examples

\ module-test.fs
\
\
include module.fs

\ Ex. 1: anonymous module

begin-module

15 constant pr1

Public:

: pu1 ." Anonymous public word " pr1 . ;

end-module


\ Ex. 2: named module Foo
module: Foo
begin-module

5 constant pr2

Public:

20 constant pu2a

: pu2 ." Foo public word " pr2 . ;

end-module


\ Ex. 3: Anonymous module using Foo

begin-module
Foo           \ depends on Foo

Public:

10 constant pu3a

: pu3 ." Anonymous public word calls " pu2 ;

end-module

\ Ex. 4: Named module Bar using Foo and anonymous

module: Bar
begin-module
Foo

pu2a pu3a + constant pr4

Public:

: pu4 ." Bar public word calls " pu3  ;

end-module

\ Ex. 5: Access private member of Bar

' Bar >private >order
cr .( Private member of Bar: ) pr4 . cr

previous


-------------------------------------------
\ Now, the interface

(      Title:  Modular programming in Forth
        File:  module.fs
     Revised:  October 5, 2011

Description:
-----------

This library provides a facility for modular programming in
Forth. It supports both named and anonymous modules, and allows
for private definitions in a module to be made accessible.
The library uses design features from several sources:
David N. Williams' root-module.fs, Bruce McFarling's suggested
module facility for the Forth Scientific Library [FSL], the
original module facility in the FSL's fsl-util auxiliary files, and
Neal Bridges' anonymous modules code.


The layout of a module is determined by the following words:

  MODULE:  BEGIN-MODULE  PUBLIC:  PRIVATE:  END-MODULE

Here, BEGIN-MODULE is where the module begins, and END-MODULE
is where it ends.  The code area between the two is "inside" the
module. The words PUBLIC: and PRIVATE: control the visibility
of the module code to external code and to the user. The use of
MODULE: , to provide a named reference to the module, is
optional.


Terminology:
-----------

1. The term "anonymous module" is a module which is
   unnamed.


Design elements:
---------------

1. A module may or may not be named:

   a] For an unnamed [anonymous] module, the public definitions
      are compiled into the current compilation wordlist.

   b] For a named module, the public definitions are compiled
      into a new wordlist. The module name may be used
      to add the public wordlist to the search order, for use
      by other modules or external code.

   c] For both a named and anonymous module, a new private
      wordlist is created. For named modules, the private
      wordlist id can be retrieved using >PRIVATE on the
      module xt. The private wordlist of an anonymous module
      is not accessible outside the module.

2. The module layout is not nestable.

3. The search order depth and compilation wordlist just before
   BEGIN-MODULE are restored by END-MODULE.

4. PRIVATE: and PUBLIC: should occur only inside a module, and may
   be invoked any number of times there, in any order.  Definitions
   inside a module usually go into its public or private
   wordlist.

5. The part of the search order present just before the
   beginning of a module may not be changed inside the module.

   [This could be made safer by actually saving and restoring
   the initial search order.  This implementation does not do
   that.]
)


\ *** GENERAL USE

[UNDEFINED] drops [IF]                 \ BMcF
: drops ( +n -- ) 0 ?DO drop LOOP ; [THEN]

[UNDEFINED] order-drops [IF]
: order-drops  ( +n -- ) ( o: wid_n ... wid_1 -- )
  0 ?DO previous LOOP ; [THEN]

[UNDEFINED] order-depth [IF]
: order-depth  ( -- n )  get-order dup >r drops r> ; [THEN]

[UNDEFINED] >order [IF]      \ same as BMcF's ADD-WORDLIST
: >order  ( wid -- order: wid )
  >r get-order r> swap 1+ set-order ; [THEN]

[UNDEFINED] order> [IF]  \ not used in this module
: order>  ( o: wid -- ) ( -- wid )
  get-order swap >r 1- set-order r> ; [THEN]

[UNDEFINED] order@ [IF]  \ not used in this module
: order@  ( o: wid -- ) ( -- wid )
  get-order over >r drops r> ; [THEN]


VARIABLE initial-defs         get-current initial-defs !
VARIABLE initial-order-depth
VARIABLE private-defs         get-current private-defs !
VARIABLE public-defs          initial-defs @ public-defs !
VARIABLE named-module         false named-module !

\ Named module

[UNDEFINED] MODULE: [IF]
\ define an extended vocabulary word for a named module
: MODULE:  ( "name" -- )
    get-current  initial-defs !
    true         named-module !
    \ create the private and public wordlists
    wordlist wordlist 2dup CREATE , ,
    set-current  private-defs !
  DOES>  ( o: -- wid )  @ >order  \ add public wordlist to search
order
;
[THEN]

\ Return private wordlist id from the xt of a named module
: >PRIVATE ( xtmodule -- wid )  >body cell+ @ ;

\ *** PUBLIC DEFINITIONS

\ public-defs @ set-current  \ same as initial-defs

\ Defns of PRIVATE: and PUBLIC: are equivalent to BMcF's
: PRIVATE:  ( -- )  private-defs @ set-current ;
: PUBLIC:   ( -- )  public-defs  @ set-current ;

\ BEGIN-MODULE and END-MODULE are not to be nested.

: BEGIN-MODULE  ( -- ) ( o: -- public private )
(
Save information that allows END-MODULE to restore the initial
search order and compilation wordlist, assuming the initial
order has remained unchanged.  Define "name" as a vocabulary
word belonging to the MODULES wordlist, and add its wordlist,
"public", to the search order.  Create an anonymous wordlist,
"private", add it to the search order, and make it the
compilation wordlist.  Store the public and private wids for use
by PUBLIC: and PRIVATE:.
)
  order-depth  initial-order-depth !
  get-current  dup >order public-defs !
  named-module @ if
    private-defs @
  else
    public-defs @ initial-defs !
    wordlist dup private-defs !
  then
  dup >order
  set-current    \ default section is private
;

: END-MODULE  ( o: <extras> -- )
(
Restore the initial search order depth and compilation wordlist.  An
ambiguous condition exists if the initial search order has been
clobbered.
)
  initial-defs @ set-current
  order-depth initial-order-depth @ - order-drops
  false  named-module ! ;


\ --------------------------------------------

\ The examples below assume that module.fs is loaded.

\ EXAMPLE 1: Layout for anonymous module
0 [IF]

BEGIN-MODULE
\ Add named modules and wordlists to the search order on which
\ this module depends.  They will be removed from the top of the
\ search order By END-MODULE.  Keeping the list of modules on
\ which it depends inside a module more clearly documents its
\ dependencies, whereas adding modules before BEGIN-MODULE:
\ makes it easier to introduce inadvertent, unwanted
\ dependencies.

<mod_1> ... <mod_n>

\ Initial definitions are private, and placed in a hidden
\   wordlist accessible only to this module.

<initial private definitions>

PUBLIC:
\ public definitions go into the current compilation wordlist

\ any number of subsequent PRIVATE: and PUBLIC: sections follow,
\ in any order.

END-MODULE
[THEN]  \ END EXAMPLE 1 LAYOUT
\ --------------------------------------------

\ EXAMPLE 2: Layout for a named module
0 [IF]

\ Make a vocabulary for the public definitions of a
\ module named Foo.

Module: Foo

BEGIN-MODULE

\ Add named modules and wordlists to the search order on which
\ this module depends.
<mod_1> ... <mod_n>

PUBLIC:
\ public definitions go into the wordlist, Foo.

\ any number of subsequent private and public sections follow,
\ in any order.

END-MODULE
[THEN]  \ END EXAMPLE 2 LAYOUT
\ --------------------------------------------

\ To access the private definitions of Foo from another
\ module, retrieve the private wordlist of Foo, and add it
\ to the search order, e.g.
\
\   ' Foo >private >order

--------------------------------

On Oct 5, 6:03=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> Below is my bare-bones sketch of an interface for modular programming
> in Forth, adapted from the code of Bruce McFarling, David N. Williams,
> and Neal Bridges.
> ...
> \ BEGIN-MODULE and END-MODULE are not to be nested.
>
> : BEGIN-MODULE =A0( -- ) ( o: -- public private )
> (
> Save information that allows END-MODULE to restore the initial
> search order and compilation wordlist, assuming the initial
> order has remained unchanged. =A0Define "name" as a vocabulary
> word belonging to the MODULES wordlist, and add its wordlist,
> "public", to the search order. =A0Create an anonymous wordlist,
> "private", add it to the search order, and make it the
> compilation wordlist. =A0Store the public and private wids for use
> by PUBLIC: and PRIVATE:.
> )
> =A0 order-depth =A0initial-order-depth !
> =A0 get-current =A0dup >order public-defs !
> =A0 named-module @ if
> =A0 =A0 private-defs @
> =A0 else
> =A0 =A0 public-defs @ initial-defs !
> =A0 =A0 wordlist dup private-defs !
> =A0 then
> =A0 dup >order
> =A0 set-current =A0 =A0\ default section is private
> ;
>
> : END-MODULE =A0( o: <extras> -- )
> (
> Restore the initial search order depth and compilation wordlist. =A0An
> ambiguous condition exists if the initial search order has been
> clobbered.
> )
> =A0 initial-defs @ set-current
> =A0 order-depth initial-order-depth @ - order-drops
> =A0 false =A0named-module ! ;
> ...


The comments for BEGIN-MODULE were copied from David's version,
without revision, and are not accurate. Also, the comments for END-
MODULE are incomplete. Below are the revised comments.

\ BEGIN-MODULE and END-MODULE are not to be nested.

: BEGIN-MODULE  ( -- ) ( o: -- public private )
(
Save information that allows END-MODULE to restore the initial
search order and compilation wordlist, assuming the initial
order has remained unchanged. Push the public wordlist onto the
search order. If we are beginning an anonymous module, create
the private wordlist. Push the private wordlist onto the search
order and set it to be the current compilation wordlist.
)
  order-depth  initial-order-depth !
  get-current  dup >order public-defs !
  named-module @ if
    private-defs @
  else
    public-defs @ initial-defs !
    wordlist dup private-defs !
  then
  dup >order
  set-current    \ default section is private
;

: END-MODULE  ( o: <extras> -- )
(
Restore the initial search order depth and compilation wordlist. An
ambiguous condition exists if the initial search order has been
clobbered. Reset the named-module flag.
)
  initial-defs @ set-current
  order-depth initial-order-depth @ - order-drops
  false  named-module ! ;

A cleaner version of module.fs. In this version MODULE: does not mess
with the search order at the time of creating the named module.
Changing the search order in between a CREATE ... DOES>  created a
difficulty for kForth's implementation of DOES> . The new defs of
MODULE: and BEGIN-MODULE are better organized, also.

Krishna

--

 (      Title:  Modular programming in Forth
        File:  module.fs
     Revised:  October 5, 2011

Description:
-----------

This library provides a facility for modular programming in
Forth. It supports both named and anonymous modules, and allows
for private definitions in a module to be made accessible.
The library uses design features from several sources:
David N. Williams' root-module.fs [v 0.8.2], Bruce McFarling's
suggested module facility for the Forth Scientific Library [FSL],
the original module facility in the FSL's fsl-util auxiliary files,
and Neal Bridges' anonymous modules code.


The layout of a module is determined by the following words:

  MODULE:  BEGIN-MODULE  PUBLIC:  PRIVATE:  END-MODULE

Here, BEGIN-MODULE is where the module begins, and END-MODULE
is where it ends.  The code area between the two is "inside" the
module. The words PUBLIC: and PRIVATE: control the visibility
of the module code to external code and to the user. The use of
MODULE: , to provide a named reference to the module, is
optional.


Terminology:
-----------

1. The term "anonymous module" is a module which is
   unnamed.


Design elements:
---------------

1. A module may or may not be named:

   a] For an unnamed [anonymous] module, the public definitions
      are compiled into the current compilation wordlist.

   b] For a named module, the public definitions are compiled
      into a new wordlist. The module name may be used
      to add the public wordlist to the search order, for use
      by other modules or external code.

   c] For both a named and anonymous module, a new private
      wordlist is created. For named modules, the private
      wordlist id can be retrieved using >PRIVATE on the
      module xt. The private wordlist of an anonymous module
      is not accessible outside the module.

2. The module layout is not nestable.

3. The search order depth and compilation wordlist just before
   BEGIN-MODULE are restored by END-MODULE.

4. PRIVATE: and PUBLIC: should occur only inside a module, and may
   be invoked any number of times there, in any order.  Definitions
   inside a module usually go into its public or private
   wordlist.

5. The part of the search order present just before the
   beginning of a module may not be changed inside the module.

   [This could be made safer by actually saving and restoring
   the initial search order.  This implementation does not do
   that.]
)


\ *** GENERAL USE

[UNDEFINED] drops [IF]                 \ BMcF
: drops ( +n -- ) 0 ?DO drop LOOP ; [THEN]

[UNDEFINED] order-drops [IF]
: order-drops  ( +n -- ) ( o: wid_n ... wid_1 -- )
  0 ?DO previous LOOP ; [THEN]

[UNDEFINED] order-depth [IF]
: order-depth  ( -- n )  get-order dup >r drops r> ; [THEN]

[UNDEFINED] >order [IF]      \ same as BMcF's ADD-WORDLIST
: >order  ( wid -- order: wid )
  >r get-order r> swap 1+ set-order ; [THEN]

[UNDEFINED] order> [IF]  \ not used in this module
: order>  ( o: wid -- ) ( -- wid )
  get-order swap >r 1- set-order r> ; [THEN]

[UNDEFINED] order@ [IF]  \ not used in this module
: order@  ( o: wid -- ) ( -- wid )
  get-order over >r drops r> ; [THEN]


[UNDEFINED] MODULE: [IF]

VARIABLE initial-defs
VARIABLE initial-order-depth
VARIABLE private-defs
VARIABLE public-defs

VARIABLE named-module         false named-module !

\ Named module

: MODULE:  ( "name" -- )
    get-current  initial-defs !
    true         named-module !
    \ create the private and public wordlists
    wordlist wordlist 2dup CREATE , ,
    public-defs !  private-defs !
  DOES>  ( o: -- wid )  @ >order  \ add public wordlist to search
order
;

\ Return private wordlist id from the xt of a named module
: >PRIVATE ( xtmodule -- wid )  >body cell+ @ ;

\ *** PUBLIC DEFINITIONS

\ public-defs @ set-current  \ same as initial-defs

\ Defns of PRIVATE: and PUBLIC: are equivalent to BMcF's
: PRIVATE:  ( -- )  private-defs @ set-current ;
: PUBLIC:   ( -- )  public-defs  @ set-current ;

\ BEGIN-MODULE and END-MODULE are not to be nested.

: BEGIN-MODULE  ( -- ) ( o: -- public private )
(
Save information that allows END-MODULE to restore the initial
search order and compilation wordlist, assuming the initial
order has remained unchanged. If we are beginning an anonymous
module, create the private wordlist. Push the public and private
wordlists onto the search order, and set the private wordlist
to be the current compilation wordlist.
)
  order-depth  initial-order-depth !
  named-module @ if
    private-defs @
    public-defs  @
    \ initial-defs already set
  else
    wordlist    dup  private-defs !
    get-current dup  public-defs !
                dup  initial-defs !
  then
  >order
  dup >order
  set-current    \ default section is private
;

: END-MODULE  ( o: <extras> -- )
(
Restore the initial search order depth and compilation wordlist. An
ambiguous condition exists if the initial search order has been
clobbered. Reset the named-module flag.
)
  initial-defs @ set-current
  order-depth initial-order-depth @ - order-drops
  false  named-module ! ;

[THEN]
\ --------------------------------------------

\ The examples below assume that module.fs is loaded.

\ EXAMPLE 1: Layout for anonymous module
0 [IF]

BEGIN-MODULE
\ Add named modules and wordlists to the search order on which
\ this module depends.  They will be removed from the top of the
\ search order By END-MODULE.  Keeping the list of modules on
\ which it depends inside a module more clearly documents its
\ dependencies, whereas adding modules before BEGIN-MODULE:
\ makes it easier to introduce inadvertent, unwanted
\ dependencies.

<mod_1> ... <mod_n>

\ Initial definitions are private, and placed in a hidden
\   wordlist accessible only to this module.

<initial private definitions>

PUBLIC:
\ public definitions go into the current compilation wordlist

\ any number of subsequent PRIVATE: and PUBLIC: sections follow,
\ in any order.

END-MODULE
[THEN]  \ END EXAMPLE 1 LAYOUT
\ --------------------------------------------

\ EXAMPLE 2: Layout for a named module
0 [IF]

\ Make a vocabulary for the public definitions of a
\ module named Foo.

Module: Foo

BEGIN-MODULE

\ Add named modules and wordlists to the search order on which
\ this module depends.
<mod_1> ... <mod_n>

PUBLIC:
\ public definitions go into the wordlist, Foo.

\ any number of subsequent private and public sections follow,
\ in any order.

END-MODULE
[THEN]  \ END EXAMPLE 2 LAYOUT
\ --------------------------------------------

\ To access the private definitions of Foo from another
\ module, retrieve the private wordlist of Foo, and add it
\ to the search order, e.g.
\
\   ' Foo >private >order

On Oct 5, 9:57=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> A cleaner version of module.fs. In this version MODULE: does not mess
> with the search order at the time of creating the named module.
> Changing the search order in between a CREATE ... DOES> =A0created a
> difficulty for kForth's implementation of DOES> . ...

Sorry, I should have said, MODULE: does not mess with the current
compilation wordlist, not the search order, at creation time. The
previous version did not mess with the search order, but did change
the current compilation wordlist in between CREATE ... DOES> , which
caused a problem in kForth, but not in Gforth. The more recent version
appears to work properly in both Forth systems.

Krishna

On Oct 5, 11:17=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 5, 9:57=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > A cleaner version of module.fs. In this version MODULE: does not mess
> > with the search order at the time of creating the named module.
> > Changing the search order in between a CREATE ... DOES> =A0created a
> > difficulty for kForth's implementation of DOES> . ...
>
> Sorry, I should have said, MODULE: does not mess with the current
> compilation wordlist, not the search order, at creation time. The
> previous version did not mess with the search order, but did change
> the current compilation wordlist in between CREATE ... DOES> , which
> caused a problem in kForth, but not in Gforth. The more recent version
> appears to work properly in both Forth systems.

An alternate approach that simplifies MODULE: could follow the PACKAGE
approach ~ define MODULE: as a WORDLIST: inside MODULES if it does not
yet exist, then go back and execute it, then DEFINITIONS to make it
the compilation wordlist.

Since WORDLIST: would be executed, and then the wordlist it created
would be executed, the problem of changing search order between CREATE
and DOES> would not arise.

Then BEGIN-MODULE does not have to be two different words hiding under
the same name, doing one thing if used after MODULE: and doing
something else if not used after MODULE:

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> Below is my bare-bones sketch of an interface for modular programming
> in Forth, adapted from the code of Bruce McFarling, David N. Williams,
> and Neal Bridges. I think it provides the essential features necessary
> for modular programming, with the exceptions of a wordlist to collect
> the module references, and Bruce's HELPER words. The code is very
> preliminary, and my testing has been only cursory, using the test
> examples given below. Feel free to offer criticisms and suggestions.

OK, I will.  

1.  This is bonkers overkill for such a simple task.

2.  Module systems for Forth have been proposed before, and have never
been popular.  This is, I suspect, because they are a PITA for
interactive development.  In practice, most programming is
maintenance, and most maintenance is interacting with and exploring an
existing system.  Having to open and close modules to explore a system
is a huge and frustrating time sink.  How do you open and close
modules, anyway?  You don't say.

Andrew.

On Oct 6, 5:17=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > Below is my bare-bones sketch of an interface for modular programming
> > in Forth, adapted from the code of Bruce McFarling, David N. Williams,
> > and Neal Bridges. I think it provides the essential features necessary
> > for modular programming, with the exceptions of a wordlist to collect
> > the module references, and Bruce's HELPER words. The code is very
> > preliminary, and my testing has been only cursory, using the test
> > examples given below. Feel free to offer criticisms and suggestions.
>
> OK, I will. =A0
>
> 1. =A0This is bonkers overkill for such a simple task.
>

Not sure our understanding of the "task" is the same. But feel free to
suggest an alternative.

> 2. =A0Module systems for Forth have been proposed before, and have never
> been popular. =A0This is, I suspect, because they are a PITA for
> interactive development. =A0In practice, most programming is
> maintenance, and most maintenance is interacting with and exploring an
> existing system. =A0Having to open and close modules to explore a system
> is a huge and frustrating time sink. =A0How do you open and close
> modules, anyway? =A0You don't say.
>

Don't know exactly what you mean by "open and close" a module. I gave
examples of usage in the first post. For a named module, execute the
module name to add its public wordlist to the search order. That
"opens" access to all of the public definitions of the named module.
For an anonymous module, the public words are already accessible
unless the search order has been changed.

To "close" a named module, i.e. make its public definitions no longer
visible to the compiler/interpreter, simply remove its public wordlist
from the search order. Consider a named module, with the name "Foo".

Foo  \ "opens" the module

\ use public defs from Foo

previous  \ "close" the module


To place both the public and private wordlists of Foo into the search
order:

Foo
' Foo >private >order

and to remove the two wordlists from the search order,

previous
previous


"Closing" a module is unnecessary when the module (public and/or
private) is used inside of any BEGIN-MODULE and END-MODULE section.

Krishna

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> On Oct 6, 5:17?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>> > Below is my bare-bones sketch of an interface for modular programming
>> > in Forth, adapted from the code of Bruce McFarling, David N. Williams,
>> > and Neal Bridges. I think it provides the essential features necessary
>> > for modular programming, with the exceptions of a wordlist to collect
>> > the module references, and Bruce's HELPER words. The code is very
>> > preliminary, and my testing has been only cursory, using the test
>> > examples given below. Feel free to offer criticisms and suggestions.
>>
>> OK, I will. 
>>
>> 1. This is bonkers overkill for such a simple task.
> 
> Not sure our understanding of the "task" is the same. But feel free to
> suggest an alternative.

As I understand it, you want modules with public and private words.

>> 2. Module systems for Forth have been proposed before, and have never
>> been popular. This is, I suspect, because they are a PITA for
>> interactive development. In practice, most programming is
>> maintenance, and most maintenance is interacting with and exploring an
>> existing system. Having to open and close modules to explore a system
>> is a huge and frustrating time sink. How do you open and close
>> modules, anyway? You don't say.
> 
> Don't know exactly what you mean by "open and close" a module. I gave
> examples of usage in the first post. For a named module, execute the
> module name to add its public wordlist to the search order. That
> "opens" access to all of the public definitions of the named module.

Most programming is maintenance programming.  Forth is excellent for
this, because its interactivity makes it easy to explore a system.
When you explore a Forth program, you need to be able to locate words
and test them interactively.  When doing that with a "modular" Forth
system, you have to be able to open modules in order to interact with
their private words, which you need to be able to do to extend and
change them.

> For an anonymous module, the public words are already accessible
> unless the search order has been changed.
> 
> To "close" a named module, i.e. make its public definitions no longer
> visible to the compiler/interpreter, simply remove its public wordlist
> from the search order. Consider a named module, with the name "Foo".
> 
> Foo  \ "opens" the module
> 
> \ use public defs from Foo
> 
> previous  \ "close" the module
> 
> 
> To place both the public and private wordlists of Foo into the search
> order:
> 
> Foo
> ' Foo >private >order
> 
> and to remove the two wordlists from the search order,
> 
> previous
> previous

And you don't think this is absurdly clumsy?  Think about how often a
maintenance programmr will have to do this in a program that uses
several modules.

Andrew.

On Oct 6, 8:42=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > On Oct 6, 5:17?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> > wrote:
> >> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> >> > Below is my bare-bones sketch of an interface for modular programmin=
g
> >> > in Forth, adapted from the code of Bruce McFarling, David N. William=
s,
> >> > and Neal Bridges. I think it provides the essential features necessa=
ry
> >> > for modular programming, with the exceptions of a wordlist to collec=
t
> >> > the module references, and Bruce's HELPER words. The code is very
> >> > preliminary, and my testing has been only cursory, using the test
> >> > examples given below. Feel free to offer criticisms and suggestions.
>
> >> OK, I will.
>
> >> 1. This is bonkers overkill for such a simple task.
>
> > Not sure our understanding of the "task" is the same. But feel free to
> > suggest an alternative.
>
> As I understand it, you want modules with public and private words.
>
> >> 2. Module systems for Forth have been proposed before, and have never
> >> been popular. This is, I suspect, because they are a PITA for
> >> interactive development. In practice, most programming is
> >> maintenance, and most maintenance is interacting with and exploring an
> >> existing system. Having to open and close modules to explore a system
> >> is a huge and frustrating time sink. How do you open and close
> >> modules, anyway? You don't say.
>
> > Don't know exactly what you mean by "open and close" a module. I gave
> > examples of usage in the first post. For a named module, execute the
> > module name to add its public wordlist to the search order. That
> > "opens" access to all of the public definitions of the named module.
>
> Most programming is maintenance programming. =A0Forth is excellent for
> this, because its interactivity makes it easy to explore a system.
> When you explore a Forth program, you need to be able to locate words
> and test them interactively. =A0When doing that with a "modular" Forth
> system, you have to be able to open modules in order to interact with
> their private words, which you need to be able to do to extend and
> change them.
>
>
>
> > For an anonymous module, the public words are already accessible
> > unless the search order has been changed.
>
> > To "close" a named module, i.e. make its public definitions no longer
> > visible to the compiler/interpreter, simply remove its public wordlist
> > from the search order. Consider a named module, with the name "Foo".
>
> > Foo =A0\ "opens" the module
>
> > \ use public defs from Foo
>
> > previous =A0\ "close" the module
>
> > To place both the public and private wordlists of Foo into the search
> > order:
>
> > Foo
> > ' Foo >private >order
>
> > and to remove the two wordlists from the search order,
>
> > previous
> > previous
>
> And you don't think this is absurdly clumsy? =A0Think about how often a
> maintenance programmr will have to do this in a program that uses
> several modules.
>
> Andrew.


Modular programming involves a trade-off between gaining a capability
for slightly more complexity in use. When one is working with a Forth
file that involves hundreds of constants, for example, a module to
isolate them in the search order is pretty handy. And, no, I don't
think the above access method is absurdly clumsy. One only has to type
the name of a module to access any of its public words. How much
simpler can it be? Access to the private words is clumsy by design,
as, typically, the programmer does not want the normal user to poke
around/modify the private defs (that's why they're private!). But,
good grief, Andrew, this is Forth! Any competent maintenance
programmer would do the following:

: open-sesame ( "name" -- ) ' dup execute >private >order ;

open-sesame Foo

There, ... the module Foo has been completely exposed (both private
and public) to the maintenance programmer.

For more desperate situations, one can bypass the module system
entirely, without changes to the code, by replacing the following defs
with suitable dummy defs :

MODULE:
BEGIN-MODULE
END-MODULE
PUBLIC:
PRIVATE:
>PRIVATE


Krishna

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> On Oct 6, 8:42?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>>
>> Most programming is maintenance programming. ?Forth is excellent for
>> this, because its interactivity makes it easy to explore a system.
>> When you explore a Forth program, you need to be able to locate words
>> and test them interactively. ?When doing that with a "modular" Forth
>> system, you have to be able to open modules in order to interact with
>> their private words, which you need to be able to do to extend and
>> change them.
>>
>> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>> > For an anonymous module, the public words are already accessible
>> > unless the search order has been changed.
>>
>> > To "close" a named module, i.e. make its public definitions no longer
>> > visible to the compiler/interpreter, simply remove its public wordlist
>> > from the search order. Consider a named module, with the name "Foo".
>>
>> > Foo ?\ "opens" the module
>>
>> > \ use public defs from Foo
>>
>> > previous ?\ "close" the module
>>
>> > To place both the public and private wordlists of Foo into the search
>> > order:
>>
>> > Foo
>> > ' Foo >private >order
>>
>> > and to remove the two wordlists from the search order,
>>
>> > previous
>> > previous
>>
>> And you don't think this is absurdly clumsy? ?Think about how often a
>> maintenance programmr will have to do this in a program that uses
>> several modules.
> 
> Modular programming involves a trade-off between gaining a capability
> for slightly more complexity in use. When one is working with a Forth
> file that involves hundreds of constants, for example, a module to
> isolate them in the search order is pretty handy. And, no, I don't
> think the above access method is absurdly clumsy. One only has to type
> the name of a module to access any of its public words. How much
> simpler can it be?

As a maintenance programmer exploring a system I don't care very much
whether a word is public or private: I need to access those words,
whatever their visibility, in order to learn about the system.  I need
to learn about the system in order to extend or change it.

This is not unusual.  This is normal.

> Access to the private words is clumsy by design, as, typically, the
> programmer does not want the normal user

What is a "normal user" ?

> to poke around/modify the private defs (that's why they're
> private!).

Of course you do.  Any program is extended and changed or it dies.
Make a program hard to extend or change and you're guaranteeing its
death.  Hide its private tools and you make it harder to change.

> But, good grief, Andrew, this is Forth! Any competent maintenance
> programmer would do the following:
> 
> : open-sesame ( "name" -- ) ' dup execute >private >order ;
> 
> open-sesame Foo
> 
> There, ... the module Foo has been completely exposed (both private
> and public) to the maintenance programmer.

So for every definition you want to try, you'll have to find out which
module it's a member of and type

open-sesame Foo

I can't believe that anyone seriously suggesting such a system has
ever done a day's maintenance work in their life.

> For more desperate situations, one can bypass the module system
> entirely, without changes to the code, by replacing the following defs
> with suitable dummy defs :
> 
> MODULE:
> BEGIN-MODULE
> END-MODULE
> PUBLIC:
> PRIVATE:
>>PRIVATE

Indeed, assuming that this doesn't cause the whole system to break.

Andrew.

On Oct 6, 11:19=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > On Oct 6, 8:42?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> > wrote:
>
> >> Most programming is maintenance programming. ?Forth is excellent for
> >> this, because its interactivity makes it easy to explore a system.
> >> When you explore a Forth program, you need to be able to locate words
> >> and test them interactively. ?When doing that with a "modular" Forth
> >> system, you have to be able to open modules in order to interact with
> >> their private words, which you need to be able to do to extend and
> >> change them.
>
> >> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> >> > For an anonymous module, the public words are already accessible
> >> > unless the search order has been changed.
>
> >> > To "close" a named module, i.e. make its public definitions no longe=
r
> >> > visible to the compiler/interpreter, simply remove its public wordli=
st
> >> > from the search order. Consider a named module, with the name "Foo".
>
> >> > Foo ?\ "opens" the module
>
> >> > \ use public defs from Foo
>
> >> > previous ?\ "close" the module
>
> >> > To place both the public and private wordlists of Foo into the searc=
h
> >> > order:
>
> >> > Foo
> >> > ' Foo >private >order
>
> >> > and to remove the two wordlists from the search order,
>
> >> > previous
> >> > previous
>
> >> And you don't think this is absurdly clumsy? ?Think about how often a
> >> maintenance programmr will have to do this in a program that uses
> >> several modules.
>
> > Modular programming involves a trade-off between gaining a capability
> > for slightly more complexity in use. When one is working with a Forth
> > file that involves hundreds of constants, for example, a module to
> > isolate them in the search order is pretty handy. And, no, I don't
> > think the above access method is absurdly clumsy. One only has to type
> > the name of a module to access any of its public words. How much
> > simpler can it be?
>
> As a maintenance programmer exploring a system I don't care very much
> whether a word is public or private: I need to access those words,
> whatever their visibility, in order to learn about the system. =A0I need
> to learn about the system in order to extend or change it.
>
> This is not unusual. =A0This is normal.
>
> > Access to the private words is clumsy by design, as, typically, the
> > programmer does not want the normal user
>
> What is a "normal user" ?
>
> > to poke around/modify the private defs (that's why they're
> > private!).
>
> Of course you do. =A0Any program is extended and changed or it dies.
> Make a program hard to extend or change and you're guaranteeing its
> death. =A0Hide its private tools and you make it harder to change.
>
> > But, good grief, Andrew, this is Forth! Any competent maintenance
> > programmer would do the following:
>
> > : open-sesame ( "name" -- ) ' dup execute >private >order ;
>
> > open-sesame Foo
>
> > There, ... the module Foo has been completely exposed (both private
> > and public) to the maintenance programmer.
>
> So for every definition you want to try, you'll have to find out which
> module it's a member of and type
>
> open-sesame Foo
>
> I can't believe that anyone seriously suggesting such a system has
> ever done a day's maintenance work in their life.
>

Interesting... so you believe it should be possible to do maintenance
programming without familiarity with the actual code. If you can
devise such a system, I'm sure the C and C++ programmers would be
beating at your door ...

In any case, Andrew, I sense that you're not particularly interested
in modular programming in Forth. Thanks for your input.

Krishna

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:

> Interesting... so you believe it should be possible to do maintenance
> programming without familiarity with the actual code. 

Every day, thousands of people are given systems they've never seen
before and told to fix them.  You have to start somewhere.

> If you can devise such a system, I'm sure the C and C++ programmers
> would be beating at your door ...
> 
> In any case, Andrew, I sense that you're not particularly interested
> in modular programming in Forth. Thanks for your input.

I'm not remotely interested in any system that makes life harder for
working programmers.  Surely it must be possible to get the modularity
you want without making extra work.

Andrew.

On Oct 6, 9:42=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Most programming is maintenance programming. =A0Forth is excellent for
> this, because its interactivity makes it easy to explore a system.
> When you explore a Forth program, you need to be able to locate words
> and test them interactively. =A0When doing that with a "modular" Forth
> system, you have to be able to open modules in order to interact with
> their private words, which you need to be able to do to extend and
> change them.

In maintaining a multi-target library where regression testing
requires compiling it on multiple platforms, the flipside of that cost
is the benefit of avoiding name conflicts.

Still, even where that benefit is substantial, more broad sharing of
external Forth source libraries will still lead to the need to patch,
modify, or extend an existing code base, which is often best done by
loading the original and then applying the patch, modification or
extension.

The approach that minimizes the cost of re-opening the private
wordlist to patch, extend or modify an existing module is:

(1) Each module inherits the PUBLIC compilation wordlist from the
wordlist in force when it begins

(2) The private wordlist of each module receives the module name, and
is stored in a single, stable wordlist.

(3) The named private wordlist acts as both a marker that the module
has been loaded, and when executed, adds itself to the search order.

(4) For the word that creates the named private wordlist, when the
module name already exists, rather than creating a new private
wordlist, the already existing wordlist is added to the search order
and made the private wordlist ~ just as with SwiftForth's packages.

On Oct 6, 2:14=A0pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > Interesting... so you believe it should be possible to do maintenance
> > programming without familiarity with the actual code.
>
> Every day, thousands of people are given systems they've never seen
> before and told to fix them. =A0You have to start somewhere.
>

I was once given a stack of several hundred pages of assembly code
(for a 6502 microcontroller, as I recall) and given only a coarse
description of the symptom of the problem in how its control of an oil
well drill didn't work as expected. Another engineer and I sat around
a large table, laying out the various code sections, slowly obtaining
an overview of how it worked. We had to get an understanding of the
specific low-level instructions at times to understand the code. After
a couple of days, we located the problem in a precision problem with
the integer division routine.

> > If you can devise such a system, I'm sure the C and C++ programmers
> > would be beating at your door ...
>
> > In any case, Andrew, I sense that you're not particularly interested
> > in modular programming in Forth. Thanks for your input.
>
> I'm not remotely interested in any system that makes life harder for
> working programmers. =A0Surely it must be possible to get the modularity
> you want without making extra work.
>

I'm always open to comparing other approaches :)

Krishna

On Oct 6, 2:43=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 6, 2:14=A0pm, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>
> > Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > > Interesting... so you believe it should be possible to do maintenance
> > > programming without familiarity with the actual code.
>
> > Every day, thousands of people are given systems they've never seen
> > before and told to fix them. =A0You have to start somewhere.
>
> I was once given a stack of several hundred pages of assembly code
> (for a 6502 microcontroller, as I recall) and given only a coarse
> description of the symptom of the problem in how its control of an oil
> well drill didn't work as expected. Another engineer and I sat around
> a large table, laying out the various code sections, slowly obtaining
> an overview of how it worked. We had to get an understanding of the
> specific low-level instructions at times to understand the code. After
> a couple of days, we located the problem in a precision problem with
> the integer division routine.
> ...

Incidentally, it's worth mentioning that I used Forth to do a
simulation of the problem with the 32-bit integer division routine (on
the 8-bit microcontroller), by writing a quick and dirty assembler/
simulator for the 6502 instructions on a PC.

Krishna

On 10/5/11 11:17 PM, Krishna Myneni wrote:
 > On Oct 5, 9:57 pm, Krishna Myneni<krishna.myn...@ccreweb.org>  wrote:
 >> A cleaner version of module.fs. In this version MODULE: does not mess
 >> with the search order at the time of creating the named module.
 >> Changing the search order in between a CREATE ... DOES>    created a
 >> difficulty for kForth's implementation of DOES>  . ...
 >
 > Sorry, I should have said, MODULE: does not mess with the current
 > compilation wordlist, not the search order, at creation time. The
 > previous version did not mess with the search order, but did change
 > the current compilation wordlist in between CREATE ... DOES>  , which
 > caused a problem in kForth, but not in Gforth. The more recent version
 > appears to work properly in both Forth systems.

I think your factorization may actually avoid a bug, or at least
a bad practice, that I've been guilty of before.  It may be the
source of the problem I'm having with root-module.fs 0.8.2 and
iForth.  I'll check into that.

Overall I like it, but I want to think a bit more about MODULE:
and BEGIN-MODULE vs. BEGIN-MODULE: and BEGIN-MODULE, and about
the private/public split within module.fs itself.

-- David

Andrew Haley <andrew29@littlepinkcloud.invalid> writes:
> As a maintenance programmer exploring a system I don't care very much
> whether a word is public or private: I need to access those words,
> whatever their visibility, in order to learn about the system.  I need
> to learn about the system in order to extend or change it. ...

That sounds like you want a debugger interface that can reach past
module boundaries.  It's a pretty standard thing to do with gdb on C
programs and so on.  It doesn't seem incompatible with letting the
normal compilation system provide more protection.

Paul Rubin <no.email@nospam.invalid> wrote:
> Andrew Haley <andrew29@littlepinkcloud.invalid> writes:
>> As a maintenance programmer exploring a system I don't care very much
>> whether a word is public or private: I need to access those words,
>> whatever their visibility, in order to learn about the system.  I need
>> to learn about the system in order to extend or change it. ...
> 
> That sounds like you want a debugger interface that can reach past
> module boundaries.  It's a pretty standard thing to do with gdb on C
> programs and so on.  It doesn't seem incompatible with letting the
> normal compilation system provide more protection.

Perhaps so: for this stuff to be practical some sort of module-aware
debugger is needed.  I suspect that's why Forth module systems aren't
much used: whatever advantage they may have is swamped by the
additional barriers they impose.

Andrew.

Using the regular VOCABULARY method, it seems obvious that a word MAIN: cou=
ld define a word header within the VOCABULARY which contains the DEFINITION=
S vocabulary, unless the current VOCABULARY is FORTH, in which case a defin=
ition within FORTH is a sensible option, eliminating an error condition, an=
d allowing fast cut/paste testing.

This would achieve all the module benefits, while being simple and efficien=
t to implement. It seems a lot of structured programming constructs are ove=
r complexified partially by search orders, and partially by some unknown pr=
ogrammer requirement. Please expound on this unknown...

Cheers Jacko

On 10/6/11 11:21 PM, David N. Williams wrote:
> On 10/5/11 11:17 PM, Krishna Myneni wrote:
>> [...]
>
> I think your factorization may actually avoid a bug, or at least
> a bad practice, that I've been guilty of before. It may be the
> source of the problem I'm having with root-module.fs 0.8.2 and
> iForth. I'll check into that.

Okay, I rewrote BEGIN-MODULE: in root-module to avoid changing
the search order or the compilation wordlist between CREATE and
DOES>, and iForth now passes root-module-test.  The link for the
revised files is below, along with the changes.  This is just
for the record, not to further advocate root-module.

With that out of the way, I'll focus on your module.fs.

-- David

------------------------------

http://www.umich.edu/~williams/archive/temp/root-module-0.8.3.zip

: _WORDLIST:  ( "name" wid -- )
(
A factor for BEGIN-MODULE: which avoids changing the search
order or compilation wordlist between CREATE and DOES>.
)
   CREATE ,
DOES> ( o: -- wid ) @ >order ;


: BEGIN-MODULE:  ( "name" -- ) ( o: -- public private )
(
Save information that allows END-MODULE to restore the initial
search order and compilation wordlist, assuming the initial
order has remained unchanged.  Define "name" as a vocabulary
word belonging to the MODULES wordlist, and add its wordlist,
"public", to the search order.  Create an anonymous wordlist,
"private", add it to the search order, and make it the
compilation wordlist.  Store the public and private wids for use
by PUBLIC: and PRIVATE:.
)
   get-current initial-defs !
   modules definitions previous
   wordlist dup _wordlist: dup >order public-defs !
   wordlist dup >order private-defs !
   definitions ;   \ default section is private

On Oct 7, 8:31=A0am, "David N. Williams" <willi...@umich.edu> wrote:
> On 10/6/11 11:21 PM, David N. Williams wrote:
>
> > On 10/5/11 11:17 PM, Krishna Myneni wrote:
> >> [...]
>
> > I think your factorization may actually avoid a bug, or at least
> > a bad practice, that I've been guilty of before. It may be the
> > source of the problem I'm having with root-module.fs 0.8.2 and
> > iForth. I'll check into that.
>
> Okay, I rewrote BEGIN-MODULE: in root-module to avoid changing
> the search order or the compilation wordlist between CREATE and
> DOES>, and iForth now passes root-module-test. =A0The link for the
> revised files is below, along with the changes. =A0This is just
> for the record, not to further advocate root-module.
>
> With that out of the way, I'll focus on your module.fs.
>
> -- David
>
> ------------------------------
>
> http://www.umich.edu/~williams/archive/temp/root-module-0.8.3.zip
>
> : _WORDLIST: =A0( "name" wid -- )
> (
> A factor for BEGIN-MODULE: which avoids changing the search
> order or compilation wordlist between CREATE and DOES>.
> )
> =A0 =A0CREATE ,
> DOES> ( o: -- wid ) @ >order ;
>
> : BEGIN-MODULE: =A0( "name" -- ) ( o: -- public private )
> (
> Save information that allows END-MODULE to restore the initial
> search order and compilation wordlist, assuming the initial
> order has remained unchanged. =A0Define "name" as a vocabulary
> word belonging to the MODULES wordlist, and add its wordlist,
> "public", to the search order. =A0Create an anonymous wordlist,
> "private", add it to the search order, and make it the
> compilation wordlist. =A0Store the public and private wids for use
> by PUBLIC: and PRIVATE:.
> )
> =A0 =A0get-current initial-defs !
> =A0 =A0modules definitions previous
> =A0 =A0wordlist dup _wordlist: dup >order public-defs !
> =A0 =A0wordlist dup >order private-defs !
> =A0 =A0definitions ; =A0 \ default section is private


I'm glad that root-module.fs is now working across the tested Forth
systems. A summary of differences between my version and root-
module.fs is the following:

1. root-modules.fs creates a separate wordlist, MODULES , which is
used as the compilation wordlist for a named module. The MODULES
wordlist is added to the search order when root-module.fs is loaded.

No such separate wordlist exists in my version, module.fs. Named
modules will appear in the current compilation wordlist. I'm still
wondering whether or not a central wordlist for named modules is truly
necessary.

2. module.fs allows for anonymous modules, while root-modules.fs only
allows named modules.

3. module.fs allows for access to the private wordlist of a named
module, while root-module.fs does not allow access to the private
wordlist of a module.

If there are any other significant differences I have overlooked,
please let me know.

Krishna

On Oct 5, 11:01=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 5, 11:17=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > On Oct 5, 9:57=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > > A cleaner version of module.fs. In this version MODULE: does not mess
> > > with the search order at the time of creating the named module.
> > > Changing the search order in between a CREATE ... DOES> =A0created a
> > > difficulty for kForth's implementation of DOES> . ...
>
> > Sorry, I should have said, MODULE: does not mess with the current
> > compilation wordlist, not the search order, at creation time. The
> > previous version did not mess with the search order, but did change
> > the current compilation wordlist in between CREATE ... DOES> , which
> > caused a problem in kForth, but not in Gforth. The more recent version
> > appears to work properly in both Forth systems.
>
> An alternate approach that simplifies MODULE: could follow the PACKAGE
> approach ~ define MODULE: as a WORDLIST: inside MODULES if it does not
> yet exist, then go back and execute it, then DEFINITIONS to make it
> the compilation wordlist.
>
> Since WORDLIST: would be executed, and then the wordlist it created
> would be executed, the problem of changing search order between CREATE
> and DOES> would not arise.
>
> Then BEGIN-MODULE does not have to be two different words hiding under
> the same name, doing one thing if used after MODULE: and doing
> something else if not used after MODULE:

I'm still wondering if the MODULES wordlist is absolutely necessary. I
understand your concern about modules becoming scattered and hard to
find if they don't appear in a centralized place. But, the natural
design flow of a modular program may avoid that problem anyway, since
END-MODULE always restores both the current compilation wordlist and
the search order depth. We implicitly assume that module code does not
change the previously existing search order other than to add to it.
That is, there may not be a problem with simply adding the module
names to the current compilation wordlist.

Also, I agree there are two words hiding in BEGIN-MODULE , but this
can be solved with factoring the two words out of BEGIN-MODULE . The
conditional call, based on the flag NAMED-MODULE , would still be used
within BEGIN-MODULE.

Krishna

On Oct 7, 8:13=A0am, jacko <jackokr...@gmail.com> wrote:
> Using the regular VOCABULARY method, it seems obvious that a word MAIN: c=
ould define a word header within the VOCABULARY which contains the DEFINITI=
ONS vocabulary, unless the current VOCABULARY is FORTH, in which case a def=
inition within FORTH is a sensible option, eliminating an error condition, =
and allowing fast cut/paste testing.
>
> This would achieve all the module benefits, while being simple and effici=
ent to implement. It seems a lot of structured programming constructs are o=
ver complexified partially by search orders, and partially by some unknown =
programmer requirement.

> Please expound on this unknown...

Ditto... i.e., can you provide a code example of how you would use
VOCABULARY and the proposed MAIN: to write a module.

Krishna

VOCABULARY new-mod DEFINITIONS

: .... ( private) ;

MAIN: exported-name ... ; ( export to the previously active vocab )

( 

And then it is just a matter of building vocabularies and maybe using some short alias names )

Cheers Jacko

And as finding the previous definitions vocab will enter as a factor of MAIN: then END-MOD could be written using it, by rolling back one vocab down the CURRENT list. : MOD VOCABULARY DEFINITIONS FORTH ; would also be useful.

In terms of un-named modules, this would not be relevant advisable with a vocab based system, as the vocab tree needs indexing. It would also be fully nestable.

Cheers Jacko

: VOPEN CONTEXT @ VOCABULARY DEFINITIONS ;
: (VCLOSE) CURRENT @ DUP @ CURRENT ! ;
: EXPORT: (VCLOSE) : CURRENT ! ;
: VCLOSE (VCLOSE) DROP CONTEXT ! ;
: IMPORT CONTEXT @ DUP @ CONTEXT ! FIND , CONTEXT ! ; IMMEDIATE


has a lot of dependencies based on the vocabulary chain and implementation of current and context. It also requires termination of the list of vocabularies by a self reference.

Cheers Jacko

On Oct 7, 11:24=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 5, 11:01=A0pm, BruceMcF <agil...@netscape.net> wrote: ...

> I'm still wondering if the MODULES wordlist is absolutely necessary. I
> understand your concern about modules becoming scattered and hard to
> find if they don't appear in a centralized place.

Of course its not absolutely necessary, but then, the module system
itself is not absolutely necessary. While it facilitates the loading
of external libraries, and makes it easy for distinct external
libraries to ensure that they do not get in each other's way with
their public interface words, and makes for a clear distinction what
is a major revision and what is a minor revision ...

.... that all *can* be done in a process of hand integration of the
library.

The primary purpose of a named MODULES wordlist remains the same as
the primary purpose of an anonymous module wordlist ~ an external
library should not impose its modular programming system on its host.

The rationale for having it in a named wordlist rather than an
anonymous wordlist is the same as for having any named module wordlist
rather than anonymous module wordlist ~ to make it easily available to
be shared by those who wish to make use of it, including patching,
modifying or extending it.

The rationale for then storing named modules in that wordlist is
because, since MODULES is always brought into the search order when
module compilation begins, it ensures that the available modules are
always available in the search order in the context of the MODULES
system being in use, while they are not cluttering up the host
wordspace when the MODULES system is not in use.

> But, the natural
> design flow of a modular program may avoid that problem anyway, since
> END-MODULE always restores both the current compilation wordlist and
> the search order depth. We implicitly assume that module code does not
> change the previously existing search order other than to add to it.

> That is, there may not be a problem with simply adding the module
> names to the current compilation wordlist.

> Also, I agree there are two words hiding in BEGIN-MODULE , but this
> can be solved with factoring the two words out of BEGIN-MODULE.

What is the benefit of having to solve it? Your named module semantics
are:

MODULE: "name"
BEGIN-MODULE-versionA
....
END-MODULE

and the anonymous module semantics are:

BEGIN-MODULE-versionB
....
END-MODULE

.... where "MODULE:" *never* makes sense to use on its own, without
BEGIN-MODULE-versionA, because what separates it from a normal F79-
style vocabulary is that it includes provision for a private wordlist
that is filled in by BEGIN-MODULE-versionA.

So there is no functional reason why BEGIN-MODULE-versionA shouldn't
be performed as part of MODULE:

If parts of BEGIN-MODULE-versionA and BEGIN-MODULE-versionB are
common, factor *them* out. There's a private wordspace already to hold
the public and private wid variables, so just put the common module
launch factors there.

Solving problems is fun, but not creating unnecessary problems to be
solved seems the preferable approach.

Or else, as with package, have a system in which the name is always
given, with a new module created if the module does not yet exist, and
the previous module opened up if the module already exists.

> The
> conditional call, based on the flag NAMED-MODULE , would still be used
> within BEGIN-MODULE.

Why? The entire reason for the conditional call is to *allow* the
naming to happen separately from BEGIN-MODULE-versionA ... but BEGIN-
MODULE-versionA should *always* follow ``MODULE:'', so the separation
is entirely superfluous.

On Oct 7, 9:13=A0am, jacko <jackokr...@gmail.com> wrote:
> Using the regular VOCABULARY method, it seems obvious that a word MAIN: c=
ould define a word header within the VOCABULARY which contains the DEFINITI=
ONS vocabulary, unless the current VOCABULARY is FORTH, in which case a def=
inition within FORTH is a sensible option, eliminating an error condition, =
and allowing fast cut/paste testing.
>
> This would achieve all the module benefits, while being simple and effici=
ent to implement. It seems a lot of structured programming constructs are o=
ver complexified partially by search orders, and partially by some unknown =
programmer requirement. Please expound on this unknown...

The main difference between PUBLIC: PRIVATE: approach is that MAIN: is
a compiling word, so public constants, variables, and created
constructs require extra steps or additional semantics to export.

But if you have MAIN: as a simple context switch, then you have the
context switch back to the private vocabulary. In a wordlist-constant
system, that is just:

this-wordlist SET-CURRENT

.... but in a Vocabulary style system, its a little more involved. And
of course, if this-wordlist is ever renamed or a code segment copied
for use in another library, each of those has to be edited to match
the correct name.

Public: and Private: simplifies that, at the cost of having some form
of a Begin and End wordset.

Clearly, the *simplest* overall system is a named Private: vocabulary
and an inherited Public: wordlist (your Main: vocabulary). If the
system follows the SwiftForth protocol of creating the private
vocabulary if it does not exist and re-opening the private vocabulary
if it does, then the public interface would be simply the wordlist
containing the named private wordlists, and the Begin-Module: word.

Such considerations would then require EXPORT: ( a replacement for MAIN:) t=
o become EXPORT as a postfix similar to IMMEDIATE. This would put the last =
definition in the outer vocabulary. This is complemented by IMPORT which sw=
itches to an outer vocabulary before the search of the next word compiling =
it to the inner vocabulary, in a similar manor to [COMPILE] would do.

In a sense IMPORT has some structural inverse connection to RECURSE in rela=
tion to what words can be found vs. what vocabularies can be found.

Cheers Jacko

(In the meantime the discussion has gotten ahead of me a little
again.)

I've reviewed the discussion by Krishna and Bruce about MODULE:
and BEGIN-MODULE, vs. BEGIN-MODULE: and BEGIN-MODULE (with
possible other names for the latter functinality); and I've
tried to think about the issue of access to private wordlists.

Although the use of the flag NAMED-MODULE does make BEGIN-MODULE
in the first approach a bit complicated, I don't see a big
difference in usability of the layouts for the two schemes.

The first scheme does naturally avoid the problem of search-
order and compilation changes between CREATE and DOES>; but as
Bruce points out, that's easy to fix, and I confirmed that in
root-module 0.8.3.

As for access to private wordlists, Krishna mentioned that when
debugging or maintaining a module you can just temporarily
redefine PRIVATE: to be a no-op, or you could comment it out if
you don't want to see redefinition warnings.

Having names in order to access private wordlists does bother me
a bit.

How about this variation?  Unnamed private module wordlists that
can be explicitly ignored in favor of public definitions, before
loading, whether or not the module is named?  Something like
this:

VARIABLE private-words   private-words on

: PRIVATE:  ( -- )
   private-words @ IF private-defs @ set-current THEN
;

Then private words for any particular module could be made
accessible by invoking PRIVATE-WORDS OFF just before the module
is loaded, followed by PRIVATE-WORDS ON just after, if you want
to hide the next set of private words.

With PRIVATE-WORDS OFF, the private wordlist would be in the
search order in the scope of the module, but definitions in
private sections would go into the public wordlist.

-- David

On 10/7/11 2:36 PM, David N. Williams wrote:
 > [...]
 >
 > How about this variation? Unnamed private module wordlists that
 > can be explicitly ignored in favor of public definitions, before
 > loading, whether or not the module is named? Something like
 > this:
 >
 > VARIABLE private-words private-words on
 >
 > : PRIVATE: ( -- )
 > private-words @ IF private-defs @ set-current THEN
 > ;

For this to work properly, the default section (compilation
wordlist) should be public, not private.

-- David

On Oct 7, 1:36=A0pm, "David N. Williams" <willi...@umich.edu> wrote:
> (In the meantime the discussion has gotten ahead of me a little
> again.)
>
> I've reviewed the discussion by Krishna and Bruce about MODULE:
> and BEGIN-MODULE, vs. BEGIN-MODULE: and BEGIN-MODULE (with
> possible other names for the latter functinality); and I've
> tried to think about the issue of access to private wordlists.
>
> Although the use of the flag NAMED-MODULE does make BEGIN-MODULE
> in the first approach a bit complicated, I don't see a big
> difference in usability of the layouts for the two schemes.
>
> The first scheme does naturally avoid the problem of search-
> order and compilation changes between CREATE and DOES>; but as
> Bruce points out, that's easy to fix, and I confirmed that in
> root-module 0.8.3.
>
> As for access to private wordlists, Krishna mentioned that when
> debugging or maintaining a module you can just temporarily
> redefine PRIVATE: to be a no-op, or you could comment it out if
> you don't want to see redefinition warnings.
>
> Having names in order to access private wordlists does bother me
> a bit.
>
> How about this variation? =A0Unnamed private module wordlists that
> can be explicitly ignored in favor of public definitions, before
> loading, whether or not the module is named? =A0Something like
> this:
>
> VARIABLE private-words =A0 private-words on
>
> : PRIVATE: =A0( -- )
> =A0 =A0private-words @ IF private-defs @ set-current THEN
> ;
>
> Then private words for any particular module could be made
> accessible by invoking PRIVATE-WORDS OFF just before the module
> is loaded, followed by PRIVATE-WORDS ON just after, if you want
> to hide the next set of private words.
>
> With PRIVATE-WORDS OFF, the private wordlist would be in the
> search order in the scope of the module, but definitions in
> private sections would go into the public wordlist.
>
> -- David

The only problem that I see with this approach is that it is not
interactive. One has to edit the loader code, which isn't really a
problem, but not as real time as if one can selectively make the
private words of a single module visible.

The notion of turning off private for all modules, e.g. using dummy
defs for the module code, also points to another problem hinted at by
Bruce. Namely, names are likely to be reused in the various modules'
private wordlists. This is fine and poses no problem for compilation
of the code. Each module will use its correct reference whether or not
the public/private mechanism is on or off. However, when code hiding
is turned off, the re-used names will be masked for the user by the
most recent definition. The user can't access the particular word
associated with a given module. In this sense, modular code actually
aids debugging rather than being a hindrance, as suggested earlier.
Being able to reference the private words of a specific module, by
name, seems to me to be a useful interactive debugging feature.

Krishna

On Oct 7, 11:17=A0am, jacko <jackokr...@gmail.com> wrote:
> And as finding the previous definitions vocab will enter as a factor of M=
AIN: then END-MOD could be written using it, by rolling back one vocab down=
 the CURRENT list. : MOD VOCABULARY DEFINITIONS FORTH ; would also be usefu=
l.
>
> In terms of un-named modules, this would not be relevant advisable with a=
 vocab based system, as the vocab tree needs indexing. It would also be ful=
ly nestable.
>
> Cheers Jacko

I think use of VOCABULARY would be good if standardized in Forth.
Unfortunately, Forth 94 lacks a standard VOCABULARY, and I don't
believe one has been proposed for Forth 200x. One reason VOCABULARY is
a good idea is that it provides a naming mechanism for wordlists, so
that names appear in the search order listing, produced by ORDER.
There will still be a need for two separate wordlists to be attached
to one name: a public wordlist and a private (hidden) wordlist. AFAIK
there is no mechanism to link two vocabularies together so that they
could be tagged as a single module. However, if a standardized
definition of VOCABULARY was available, and reserved at least two
wordlists, rather than a single one, the entire module mechanism could
be more tightly integrated with the Forth system itself.

My understanding of your suggested approach is that it is similar to
an anonymous module, where the public definitions go into the current
compilation wordlist (prior to VOCABULARY), and the "private"
definitions go into the wordlist associated with the VOCABULARY. This
approach provides at least one feature that my code doesn't have,
namely an anonymous module with accessible "private" definitions.

Krishna

On Oct 7, 2:11=A0pm, jacko <jackokr...@gmail.com> wrote:
> Such considerations would then require EXPORT: ( a replacement for MAIN:)=
 to become EXPORT as a postfix similar to IMMEDIATE. This would put the las=
t definition in the outer vocabulary. This is complemented by IMPORT which =
switches to an outer vocabulary before the search of the next word compilin=
g it to the inner vocabulary, in a similar manor to [COMPILE] would do.

I don't think the standard search order words allow moving completed
definitions between wordlists, so EXPORT would have to be a current
switch.

Given the ORDER> semantics,

MyVocab order> SET-CURRENT

.... local definitions, constants, variables, etc.

EXPORT

.... exported definitions, constants, variables, etc.

MVocab order> SET-CURRENT

.... local definitions, constants, variables, etc.

EXPORT

.... exported definitions, constants, variables, etc.

>
> In a sense IMPORT has some structural inverse connection to RECURSE in re=
lation to what words can be found vs. what vocabularies can be found.

On Oct 7, 7:49=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> There will still be a need for two separate wordlists to be attached
> to one name: a public wordlist and a private (hidden) wordlist.

Why? Its perfectly workable for a substantial number of modules to
define their public words into a single common wordlist ~ there's no
necessity for each module to have a distinct named wordlist for its
public words.

So, say you have eight (or, 18 or 28) modules with their public words
all in a common vocabulary ~ if you do not like dnw's WORDLIST: for an
explicit Forth79 style vocabulary, then in the previous thread,
NAMESPACE is also used for Forth79 style vocabularies.

: USE-NAMESPACE: ( "name" -- )
   >IN @ BL WORD FIND 0=3D IF
      2DROP DUP >IN ! NAMESPACE
      >IN ! '
   ELSE NIP THEN
   EXECUTE DEFINITIONS ;

ONLY FORTH USE-NAMESPACE: FSL
INCLUDE fsl-util.x
.... (other FLS modules)

Now all the public words from FSL are in a single vocabulary.

On Oct 7, 2:36=A0pm, "David N. Williams" <willi...@umich.edu> wrote:
> Having names in order to access private wordlists does bother me
> a bit.

Why? The point of private wordlists is to avoid unnecessarily
cluttering up the main target wordlist with factors and static local
variables and such that are secondary to the main capability being
provided ~ and, indeed,not necessarily stable from one minor version
to the next.

Having them accessible by name, however, seems a natural Forth
approach to the question of how you get at the private words when it
becomes apparent later that the public interface needs to be modified
to cope with a special situation, or there is a bug that can be
patched, or that the module can be extended to meet an unanticipated
need.

On Oct 7, 11:43=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 7, 7:49=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > There will still be a need for two separate wordlists to be attached
> > to one name: a public wordlist and a private (hidden) wordlist.
>
> Why? Its perfectly workable for a substantial number of modules to
> define their public words into a single common wordlist ~ there's no
> necessity for each module to have a distinct named wordlist for its
> public words.
>
> So, say you have eight (or, 18 or 28) modules with their public words
> all in a common vocabulary ~ if you do not like dnw's WORDLIST: for an
> explicit Forth79 style vocabulary, then in the previous thread,
> NAMESPACE is also used for Forth79 style vocabularies.
>
> : USE-NAMESPACE: ( "name" -- )
> =A0 =A0>IN @ BL WORD FIND 0=3D IF
> =A0 =A0 =A0 2DROP DUP >IN ! NAMESPACE
> =A0 =A0 =A0 >IN ! '
> =A0 =A0ELSE NIP THEN
> =A0 =A0EXECUTE DEFINITIONS ;
>
> ONLY FORTH USE-NAMESPACE: FSL
> INCLUDE fsl-util.x
> ... (other FLS modules)
>
> Now all the public words from FSL are in a single vocabulary.

The suggestion of using a single common wordlist for the public words
of a number of modules is, of course, the approach being considered
for the FSL, and it is the feature provided by anonymous wordlists.
The common wordlist of course happens to be the current compilation
wordlist. But, as you point out above, it could just as well be
another wordlist.  I have nothing against the functionality of DNW's
WORDLIST: , but I think the name of the word might be problematic.
Whatever name is used to create the named wordlist, the above approach
is fine for certain uses, e.g. collecting a group of related modules
such as the FSL into a library of modules.

But, what's appropriate for the FSL may not be the preferred approach
in other cases. A library may also consist of a single module. In this
case there is no difference between creating a named module that has
both public and private wordlists, and using a WORDLIST: to create the
public wordlist separately, and declaring an anonymous module to
produce the private wordlist. The latter procedure is more tedious.

As a further example of the rationale for a module to *be able to
have* both public and private wordlists, consider one situation where
the use of modules arises naturally. We have a collection of modules
each representing a different version of the same functional module.
Different parts of an application might need to use different versions
of a module -- that's not an outlandish scenario in an operational
situation where maintenance is being performed. Then, loading all of
the public definitions of different versions of a module into a single
common wordlist would be a disaster. In this case, each module should
have its own public and private wordlist, and the public wordlist of a
specific module should be accessible by name.

I'll qualify the above example by saying that I haven't encountered
such a situation myself. However, I don't think it's preferable to box
ourselves into an overly restrictive notion of a module.

Krishna

On Oct 7, 11:58=A0am, BruceMcF <agil...@netscape.net> wrote:
> On Oct 7, 11:24=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > On Oct 5, 11:01=A0pm, BruceMcF <agil...@netscape.net> wrote: ...
> > I'm still wondering if the MODULES wordlist is absolutely necessary. I
> > understand your concern about modules becoming scattered and hard to
> > find if they don't appear in a centralized place.
>
> Of course its not absolutely necessary, but then, the module system
> itself is not absolutely necessary. While it facilitates the loading
> of external libraries, and makes it easy for distinct external
> libraries to ensure that they do not get in each other's way with
> their public interface words, and makes for a clear distinction what
> is a major revision and what is a minor revision ...
>
> ... that all *can* be done in a process of hand integration of the
> library.
>
> The primary purpose of a named MODULES wordlist remains the same as
> the primary purpose of an anonymous module wordlist ~ an external
> library should not impose its modular programming system on its host.
>

I think the word "library" is key here. To me, a library means a
grouping of related modules. And, this may be why the notion of
storing all references to named modules into a single wordlist called
MODULES, bothered me. I didn't see much benefit in placing references
into the MODULES wordlist over placing the references into the current
compilation wordlist. The only apparent benefit was avoiding potential
name collisions when searching to see if a named module already
exists. But, if the provision to group logically connected modules
into distinct libraries (which are nothing more than wordlists
containing references to the modules), that would be worthwhile IMO.
The current implementation of root-module.fs assumes a single library
called MODULES. Perhaps something like,

LIBRARY: <lib1>

MODULE: <mod1>
BEGIN-MODULE
.... END-MODULE

MODULE: <mod2>
BEGIN-MODULE
.... END-MODULE

....

LIBRARY: <lib2>

MODULE: <mod3>
....


There may be external (C) library interfaces that use the word
LIBRARY: so another appropriate name may have to be be chosen. And,
the definition of LIBRARY: is nothing more than DNW's WORDLIST: .



> The rationale for having it in a named wordlist rather than an
> anonymous wordlist is the same as for having any named module wordlist
> rather than anonymous module wordlist ~ to make it easily available to
> be shared by those who wish to make use of it, including patching,
> modifying or extending it.
>
> The rationale for then storing named modules in that wordlist is
> because, since MODULES is always brought into the search order when
> module compilation begins, it ensures that the available modules are
> always available in the search order in the context of the MODULES
> system being in use, while they are not cluttering up the host
> wordspace when the MODULES system is not in use.
>
> > But, the natural
> > design flow of a modular program may avoid that problem anyway, since
> > END-MODULE always restores both the current compilation wordlist and
> > the search order depth. We implicitly assume that module code does not
> > change the previously existing search order other than to add to it.
> > That is, there may not be a problem with simply adding the module
> > names to the current compilation wordlist.
> > Also, I agree there are two words hiding in BEGIN-MODULE , but this
> > can be solved with factoring the two words out of BEGIN-MODULE.
>
> What is the benefit of having to solve it? Your named module semantics
> are:
>
> MODULE: "name"
> BEGIN-MODULE-versionA
> ...
> END-MODULE
>
> and the anonymous module semantics are:
>
> BEGIN-MODULE-versionB
> ...
> END-MODULE
>
> ... where "MODULE:" *never* makes sense to use on its own, without
> BEGIN-MODULE-versionA, because what separates it from a normal F79-
> style vocabulary is that it includes provision for a private wordlist
> that is filled in by BEGIN-MODULE-versionA.
>
> So there is no functional reason why BEGIN-MODULE-versionA shouldn't
> be performed as part of MODULE:
>

I agree that there's no apparent *functional reason* why MODULE: has
to be split into two functions. It's maybe a matter of programming
taste and desire for consistency that I prefer to require BEGIN-MODULE
for both named and anonymous modules. Indeed the work of actually
creating the two wordlists for a named module can be pushed into BEGIN-
MODULE rather than being performed in MODULE: . This is sort of
opposite to your suggestion, but then more clearly delineates the role
of MODULE: as simply a naming mechanism.


> If parts of BEGIN-MODULE-versionA and BEGIN-MODULE-versionB are
> common, factor *them* out. There's a private wordspace already to hold
> the public and private wid variables, so just put the common module
> launch factors there.
>
> Solving problems is fun, but not creating unnecessary problems to be
> solved seems the preferable approach.
>
> Or else, as with package, have a system in which the name is always
> given, with a new module created if the module does not yet exist, and
> the previous module opened up if the module already exists.
> ...

Haven't thought about the need to add further definitions to an
existing module, elsewhere in a program... Is that a good idea?

Krishna

On Oct 8, 9:32=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:

> I think the word "library" is key here. To me, a library means a
> grouping of related modules. And, this may be why the notion of
> storing all references to named modules into a single wordlist called
> MODULES, bothered me. I didn't see much benefit in placing references
> into the MODULES wordlist over placing the references into the current
> compilation wordlist.
....

As the library up the street is a collection of books on a single
subject?

Some libraries would be a grouping of related modules. Others would be
several groupings of related modules, providing a range of
capabilities.

However, an aspect of a library like FSL is that it is a "lending
library", a means for code sharing so that capabilities can be
*shared*.

> Haven't thought about the need to add further definitions to an
> existing module, elsewhere in a program... Is that a good idea?

A given library may require tailoring for a specific system or
specific task. Sometimes that will require a modified definition for
part of the public interface.

For a public, external library, its often preferable to just load the
original library and apply a patch. The module system should make it
as straightforward as possible to apply a patch.

And of course, what the source code is designed to be used for and
what it turns out to be usable for is not always the same thing. That
is the other reason to be skeptical about having anonymous private
modules: even if there is some trick to apply to temporarily make them
available, making them available by turning off the Private mode is
only a debugging trick, its not an approach that is designed to
provide support for code sharing.

And code sharing also includes two different sources that are both
using their own selections of modules from the same library but using
it in different ways. A system that does not make is straightforward
to detect whether a given module has already been loaded and to take
appropriate action is broken. It would be especially ironic if what
makes it hard to detect and reuse code that has already been compiled
is the fact that the application loading the library source uses the
same module system as the library, which is to say in the case of
encouraging public interfaces each hidden within its own named
wordlist, the second source to be loaded having no way to know where
the library modules were loaded by the first source to be loaded.

On Oct 8, 9:06=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:

> The suggestion of using a single common wordlist for the public words
> of a number of modules is, of course, the approach being considered
> for the FSL, and it is the feature provided by anonymous wordlists.

An even more important feature is the ability of an application
loading FSL code to decide for itself whether the public words should
be in a separate wordlist at all, whether they should be in a single
separate wordlist, whether they should be in a number of distinct
wordlists on their own, or whether they should be in one or more
distinct wordlists shared with words defined by other sources.

> The common wordlist of course happens to be the current compilation
> wordlist. But, as you point out above, it could just as well be
> another wordlist. =A0I have nothing against the functionality of DNW's
> WORDLIST: , but I think the name of the word might be problematic.
> Whatever name is used to create the named wordlist, the above approach
> is fine for certain uses, e.g. collecting a group of related modules
> such as the FSL into a library of modules.

> But, what's appropriate for the FSL may not be the preferred approach
> in other cases. A library may also consist of a single module.

> In this
> case there is no difference between creating a named module that has
> both public and private wordlists, and using a WORDLIST: to create the
> public wordlist separately, and declaring an anonymous module to
> produce the private wordlist. The latter procedure is more tedious.

If the means of making a named wordspace and the modular system for
providing externally accessible and locally accessible are not welded
together, its easy to make the named namespace optional ~ for
instance, have the module in one source, and a second loadfile that
includes the module into a named namespace. Indeed, a system that does
not weld the two together can easily be provided with several distinct
loadfiles for adapting to several different systems.

> As a further example of the rationale for a module to *be able to
> have* both public and private wordlists, consider one situation where
> the use of modules arises naturally. We have a collection of modules
> each representing a different version of the same functional module.
> Different parts of an application might need to use different versions
> of a module -- that's not an outlandish scenario in an operational
> situation where maintenance is being performed.

Yes. An excellent example of the pitfalls of welding the reference to
the private namespace to the public module name, and the benefit of
being able to separate the naming of a namespace from the name of a
module ~ one it becomes clear that a "new minor version" is required,
because of changes to the private namespace have to be made, the new
private module can be given a name that reflects that ... eg,
MaxEnt-01 rather than MaxEnt.

Then the application that includes different versions of a module can
test by name for whether the correct *version* of the module is
available, if so use it, and if not, load the required version.
Because the module itself does not impose a name on its public
namespace, the loading application is free to place the two distinct
versions of the same library in appropriate places ~ which could well
be to load the code using the "old" version first, then load the code
using the "new" version, where the new version may well have
conditional compilation that takes into account the fact that the old
version may be around.

> Then, loading all of
> the public definitions of different versions of a module into a single
> common wordlist would be a disaster.

The difference between welding the naming of the public namespace with
the definition of the module and not doing so is that in the former
case, if the result is a disaster, the disaster is welded in. In the
latter case, if the first loadscript is a disaster, it can be fixed
until it isn't a disaster anymore.

> In this case, each module should
> have its own public and private wordlist, and the public wordlist of a
> specific module should be accessible by name.

> I'll qualify the above example by saying that I haven't encountered
> such a situation myself.

I'll qualify the above by saying that quite a long while ago, it was
trying to work out both modular programming and a simple versioning
system for BMW and creating an unreleasable disaster that started my
sporadic explorations of these issues.

On Oct 8, 10:11=A0am, BruceMcF <agil...@netscape.net> wrote:
> On Oct 8, 9:06=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > The suggestion of using a single common wordlist for the public words
> > of a number of modules is, of course, the approach being considered
> > for the FSL, and it is the feature provided by anonymous wordlists.
>
> An even more important feature is the ability of an application
> loading FSL code to decide for itself whether the public words should
> be in a separate wordlist at all, whether they should be in a single
> separate wordlist, whether they should be in a number of distinct
> wordlists on their own, or whether they should be in one or more
> distinct wordlists shared with words defined by other sources.
>
> > The common wordlist of course happens to be the current compilation
> > wordlist. But, as you point out above, it could just as well be
> > another wordlist. =A0I have nothing against the functionality of DNW's
> > WORDLIST: , but I think the name of the word might be problematic.
> > Whatever name is used to create the named wordlist, the above approach
> > is fine for certain uses, e.g. collecting a group of related modules
> > such as the FSL into a library of modules.
> > But, what's appropriate for the FSL may not be the preferred approach
> > in other cases. A library may also consist of a single module.
> > In this
> > case there is no difference between creating a named module that has
> > both public and private wordlists, and using a WORDLIST: to create the
> > public wordlist separately, and declaring an anonymous module to
> > produce the private wordlist. The latter procedure is more tedious.
>
> If the means of making a named wordspace and the modular system for
> providing externally accessible and locally accessible are not welded
> together, its easy to make the named namespace optional ~ for
> instance, have the module in one source, and a second loadfile that
> includes the module into a named namespace. Indeed, a system that does
> not weld the two together can easily be provided with several distinct
> loadfiles for adapting to several different systems.
>
> > As a further example of the rationale for a module to *be able to
> > have* both public and private wordlists, consider one situation where
> > the use of modules arises naturally. We have a collection of modules
> > each representing a different version of the same functional module.
> > Different parts of an application might need to use different versions
> > of a module -- that's not an outlandish scenario in an operational
> > situation where maintenance is being performed.
>
> Yes. An excellent example of the pitfalls of welding the reference to
> the private namespace to the public module name,

I don't see this at all, from my example. Why would the public and
private defns of a single module have different versions? The module
itself has an overall version. There is a collection of similar
modules, of different versions, as distinguished by their names.


> ... and the benefit of
> being able to separate the naming of a namespace from the name of a
> module ~ one it becomes clear that a "new minor version" is required,
> because of changes to the private namespace have to be made, the new
> private module can be given a name that reflects that ... eg,
> MaxEnt-01 rather than MaxEnt.
>

I'm not sure what you mean by "private module". Are you using private
in the same context as private definitions within a module?

> Then the application that includes different versions of a module can
> test by name for whether the correct *version* of the module is
> available, if so use it, and if not, load the required version.

Yes, that's what I was envisioning. For example, let's say there's a
collection of different versions of the module Foo. Each is a separate
module, e.g.

Foo-01
Foo-02
....
etc.

If a part of my application wanted to use version 1, at that point,
all that is required is to type,

Foo-01

and, in another part, if version 2 is desired,

Foo-02

and so on. All of the Foo-??s names can reside in a single wordlist.
This would be a library of different versions of Foo. I don't
understand your statement that the module is imposing its name on the
application. The application is free to choose among the available
modules, as you state below.

> Because the module itself does not impose a name on its public
> namespace, the loading application is free to place the two distinct
> versions of the same library in appropriate places ~ which could well
> be to load the code using the "old" version first, then load the code
> using the "new" version, where the new version may well have
> conditional compilation that takes into account the fact that the old
> version may be around.
>
> > Then, loading all of
> > the public definitions of different versions of a module into a single
> > common wordlist would be a disaster.
>
> The difference between welding the naming of the public namespace with
> the definition of the module and not doing so is that in the former
> case, if the result is a disaster, the disaster is welded in. In the
> latter case, if the first loadscript is a disaster, it can be fixed
> until it isn't a disaster anymore.
>
> > In this case, each module should
> > have its own public and private wordlist, and the public wordlist of a
> > specific module should be accessible by name.
> > I'll qualify the above example by saying that I haven't encountered
> > such a situation myself.
>
> I'll qualify the above by saying that quite a long while ago, it was
> trying to work out both modular programming and a simple versioning
> system for BMW and creating an unreleasable disaster that started my
> sporadic explorations of these issues.

I think the module system should be capable of being used, at least in
a simple way, for versioning. And, I see the dual wordlists of a named
module as being essential to support that.

Krishna

On Oct 8, 8:32=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 7, 11:58=A0am, BruceMcF <agil...@netscape.net> wrote:
>

> > > Also, I agree there are two words hiding in BEGIN-MODULE , but this
> > > can be solved with factoring the two words out of BEGIN-MODULE.
>
> > What is the benefit of having to solve it? Your named module semantics
> > are:
>
> > MODULE: "name"
> > BEGIN-MODULE-versionA
> > ...
> > END-MODULE
>
> > and the anonymous module semantics are:
>
> > BEGIN-MODULE-versionB
> > ...
> > END-MODULE
>
> > ... where "MODULE:" *never* makes sense to use on its own, without
> > BEGIN-MODULE-versionA, because what separates it from a normal F79-
> > style vocabulary is that it includes provision for a private wordlist
> > that is filled in by BEGIN-MODULE-versionA.
>
> > So there is no functional reason why BEGIN-MODULE-versionA shouldn't
> > be performed as part of MODULE:
>
> I agree that there's no apparent *functional reason* why MODULE: has
> to be split into two functions. It's maybe a matter of programming
> taste and desire for consistency that I prefer to require BEGIN-MODULE
> for both named and anonymous modules. Indeed the work of actually
> creating the two wordlists for a named module can be pushed into BEGIN-
> MODULE rather than being performed in MODULE: . This is sort of
> opposite to your suggestion, but then more clearly delineates the role
> of MODULE: as simply a naming mechanism.
> ...

Here's a better factoring, which makes clear that MODULE: only
provides a naming mechanism. The wordlist creation and other
initialization is now entirely within BEGIN-MODULE.

Krishna
--

(      Title:  Modular programming in Forth
        File:  module.fs
     Version:  0.2
     Revised:  October 8, 2011

Description:
-----------

This library provides a facility for modular programming in
Forth. It supports both named and anonymous modules, and allows
for private definitions in a module to be made accessible.
The library uses design features from several sources:
David N. Williams' root-module.fs [v 0.8.2], Bruce McFarling's
suggested module facility for the Forth Scientific Library [FSL],
the original module facility in the FSL's fsl-util auxiliary files,
and Neal Bridges' anonymous modules code.


The layout of a module is determined by the following words:

  MODULE:  BEGIN-MODULE  PUBLIC:  PRIVATE:  END-MODULE

Here, BEGIN-MODULE is where the module begins, and END-MODULE
is where it ends.  The code area between the two is "inside" the
module. The words PUBLIC: and PRIVATE: control the visibility
of the module code to external code and to the user. The use of
MODULE: , to provide a named reference to the module, is
optional.


Terminology:
-----------

1. The term "anonymous module" is a module which is
   unnamed.


Design elements:
---------------

1. A module may or may not be named:

   a] For an unnamed [anonymous] module, the public definitions
      are compiled into the current compilation wordlist.

   b] For a named module, the public definitions are compiled
      into a new wordlist. The module name may be used
      to add the public wordlist to the search order, for use
      by other modules or external code.

   c] For both a named and anonymous module, a new private
      wordlist is created. For named modules, the private
      wordlist id can be retrieved using >PRIVATE on the
      module xt. The private wordlist of an anonymous module
      is not accessible outside the module.

2. The module layout is not nestable.

3. The search order depth and compilation wordlist just before
   BEGIN-MODULE are restored by END-MODULE.

4. PRIVATE: and PUBLIC: should occur only inside a module, and may
   be invoked any number of times there, in any order.  Definitions
   inside a module usually go into its public or private
   wordlist.

5. The part of the search order present just before the
   beginning of a module may not be changed inside the module.

   [This could be made safer by actually saving and restoring
   the initial search order.  This implementation does not do
   that.]
)


\ *** GENERAL USE

[UNDEFINED] drops [IF]                 \ BMcF
: drops ( +n -- ) 0 ?DO drop LOOP ; [THEN]

[UNDEFINED] order-drops [IF]
: order-drops  ( +n -- ) ( o: wid_n ... wid_1 -- )
  0 ?DO previous LOOP ; [THEN]

[UNDEFINED] order-depth [IF]
: order-depth  ( -- n )  get-order dup >r drops r> ; [THEN]

[UNDEFINED] >order [IF]      \ same as BMcF's ADD-WORDLIST
: >order  ( wid -- order: wid )
  >r get-order r> swap 1+ set-order ; [THEN]

[UNDEFINED] order> [IF]  \ not used in this module
: order>  ( o: wid -- ) ( -- wid )
  get-order swap >r 1- set-order r> ; [THEN]

[UNDEFINED] order@ [IF]  \ not used in this module
: order@  ( o: wid -- ) ( -- wid )
  get-order over >r drops r> ; [THEN]


[UNDEFINED] MODULE: [IF]

VARIABLE initial-defs
VARIABLE initial-order-depth
VARIABLE private-defs
VARIABLE public-defs

VARIABLE named-module         0 named-module !

\ Create the name for a named module and reserve space
\ to store public and private wordlists
: MODULE:  ( "name" -- )
    CREATE  here named-module !  0 , 0 ,
  DOES>  ( o: -- wid )  @ >order  \ add public wordlist to search
order
;

\ Return private wordlist id from the xt of a named module
: >PRIVATE ( xtmodule -- wid )  >body cell+ @ ;

\ *** PUBLIC DEFINITIONS

\ public-defs @ set-current  \ same as initial-defs

\ Defns of PRIVATE: and PUBLIC: are equivalent to BMcF's
: PRIVATE:  ( -- )  private-defs @ set-current ;
: PUBLIC:   ( -- )  public-defs  @ set-current ;

\ BEGIN-MODULE and END-MODULE are not to be nested.

: BEGIN-MODULE  ( -- ) ( o: -- public private )
(
Save information that allows END-MODULE to restore the initial
search order and compilation wordlist, assuming the initial
order has remained unchanged. If we are beginning an anonymous
module, create the private wordlist. Push the public and private
wordlists onto the search order, and set the private wordlist
to be the current compilation wordlist.
)
  get-current  initial-defs !
  order-depth  initial-order-depth !
  wordlist      \ new private wordlist
  named-module @ if
    wordlist    \ new public wordlist
    2dup named-module @ 2!
  else
    get-current  \ use compilation wordlist
  then
  dup   >order     public-defs !
  dup   >order dup private-defs !
  set-current    \ default section is private
;

: END-MODULE  ( o: <extras> -- )
(
Restore the initial search order depth and compilation wordlist. An
ambiguous condition exists if the initial search order has been
clobbered. Reset the named-module body address.
)
  initial-defs @ set-current
  order-depth initial-order-depth @ - order-drops
  0  named-module ! ;

[THEN]
\ --------------------------------------------

\ The examples below assume that module.fs is loaded.

\ EXAMPLE 1: Layout for anonymous module
0 [IF]

BEGIN-MODULE
\ Add named modules and wordlists to the search order on which
\ this module depends.  They will be removed from the top of the
\ search order By END-MODULE.  Keeping the list of modules on
\ which it depends inside a module more clearly documents its
\ dependencies, whereas adding modules before BEGIN-MODULE:
\ makes it easier to introduce inadvertent, unwanted
\ dependencies.

<mod_1> ... <mod_n>

\ Initial definitions are private, and placed in a hidden
\   wordlist accessible only to this module.

<initial private definitions>

PUBLIC:
\ public definitions go into the current compilation wordlist

\ any number of subsequent PRIVATE: and PUBLIC: sections follow,
\ in any order.

END-MODULE
[THEN]  \ END EXAMPLE 1 LAYOUT
\ --------------------------------------------

\ EXAMPLE 2: Layout for a named module
0 [IF]

\ Make a vocabulary for the public definitions of a
\ module named Foo.

Module: Foo

BEGIN-MODULE

\ Add named modules and wordlists to the search order on which
\ this module depends.
<mod_1> ... <mod_n>

PUBLIC:
\ public definitions go into the wordlist, Foo.

\ any number of subsequent private and public sections follow,
\ in any order.

END-MODULE
[THEN]  \ END EXAMPLE 2 LAYOUT
\ --------------------------------------------

\ To access the private definitions of Foo from another
\ module, retrieve the private wordlist of Foo, and add it
\ to the search order, e.g.
\
\   ' Foo >private >order

On Oct 8, 11:39=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> I don't see this at all, from my example. Why would the public and
> private defns of a single module have different versions?

They wouldn't have different versions, but obviously a change in the
private definitions without a change in the public interface may or
may not be a change that a source using the public interface alone has
to care about ~ it might be a bugfix, for example ~ but is a change
that a source using the private namespace would have to take into
account.

> The module
> itself has an overall version. There is a collection of similar
> modules, of different versions, as distinguished by their names.

Arranged how? All available in the same namespace? Older versions
stored hierarchically within the namespace of the new version. Welding
the naming of the public namespace to the module definition limits the
flexibility in how that is handled. Even more, creating named
namespaces as vocabularies limits the flexibility in how that is
handled, in a system that relies on operators on wordlist constants
rather than operators on the top of the search order stack.

> > ... and the benefit of
> > being able to separate the naming of a namespace from the name of a
> > module ~ one it becomes clear that a "new minor version" is required,
> > because of changes to the private namespace have to be made, the new
> > private module can be given a name that reflects that ... eg,
> > MaxEnt-01 rather than MaxEnt.

> I'm not sure what you mean by "private module". Are you using private
> in the same context as private definitions within a module?

That should have been "the new private namespace can be given a name
that reflects that".

> > Then the application that includes different versions of a module can
> > test by name for whether the correct *version* of the module is
> > available, if so use it, and if not, load the required version.

> Yes, that's what I was envisioning. For example, let's say there's a
> collection of different versions of the module Foo. Each is a separate
> module, e.g.
>
> Foo-01
> Foo-02
> ...
> etc.
>
> If a part of my application wanted to use version 1, at that point,
> all that is required is to type,
>
> Foo-01
>
> and, in another part, if version 2 is desired,
>
> Foo-02
>
> and so on. All of the Foo-??s names can reside in a single wordlist.
> This would be a library of different versions of Foo. I don't
> understand your statement that the module is imposing its name on the
> application. The application is free to choose among the available
> modules, as you state below.

You are ignoring the fact that the application and/or the host system
may be used in a system that is *using* a different system for *its
own* version management. The creation of a distinct namespace for the
public interface in module source code imposes the library's model of
version management on the host system, which requires additional
bridging functions for the host system to integrate the two.

A system that is using namespaces for some form of version control can
use its intrinsic actions to include an external module that inherits
its public namespace from its caller, without requiring any bridging
or multiple version systems of different designs if it is only making
use of the public interface words.

Separation the naming of the public interface namespace from the
definition of the module makes it easy for a library to provide a
loadscript that provides a default layout for its modules, while a
system that uses an alternative system can load the public interfaces
of the underlying modules directly into its version control.

I had at one point three levels of scripts for the simple library
oversight system I was working on for retro Forth systems ~ the base
level just defined words and assumed the search order was set up as
required, so it made extensive use [UNDEFINED] and [DEFINED] and just
assumed that name conflicts which would break that ~ say, using a
Toolbelt2000 semantics where some implementation had the same name
under distinct semantics ~ were taken care of by the caller. The base
level never touched the search order. Level two was aware of the
Niclos search order management system, but no aware of versioning. And
level three loadscripts were aware of the simple version control
system.

An external library with the anonymous modules alone can called
individual module sources as level one scripts, so long as the search
order words are present (native Niclos level one scripts can be
compiled by systems without search order words at all). And with
tailored Niclos loadscripts for different versions of the modules,
could be easily integrated into the simple minded Niclos version
control system by a single level three loadscript.

Also, an external library with inherited public compilation namespace
and private wordlists entered into a dedicated namespace can be easily
integrated, with even more flexibility, since extracting the wordlist
id from MODULES and saving it as a wordlist compiler constant in the
Niclos registry is straightforward. So individual modules could be
loaded "as if" they were level one script if desired, and not worry
about making the private namespace accessible, while they could also
be loaded by a level two script, including the actions to install the
module versions of the private namespaces into the Niclos version
system.

But the external module starts to hide its public interface into
namespaces that it names, and things start to get more complex. The
Niclos system is not built around the concept of public interfaces
being compiled into distinct separate namespaces by functional group,
but around public interfaces being compiled into as large a common
namespace as practicable. So any individual modules from that
individual library needs a level two script that loads the module,
then opens the module and the actual target wordlist together, then
defines the module public interface words into the actual target
wordlist in a series of:
   : }}Condition }}Condition ;
   ...
.... definitions.

> I think the module system should be capable of being used, at least in
> a simple way, for versioning. And, I see the dual wordlists of a named
> module as being essential to support that.

For the public interface, it should be offered to the host system, but
not imposed on the host system.

The goals should include not *just* capable of being used that way,
but also capable of *not* being used that way, if a alternate or more
capable version control system is already in place in the host system.

For the private namespaces, its entirely reasonable to have an
approach to naming the namespace that entails using the modular
control operations used by the library ~ a user of a private namespace
needs to be more aware of how the library code is put together than
someone who only uses the public interfaces.

In article <3c509113-57fc-4f90-9ea0-7c5ddcfaa63b@h10g2000yqd.googlegroups.com>,
Krishna Myneni  <krishna.myneni@ccreweb.org> wrote:
>
>I think use of VOCABULARY would be good if standardized in Forth.
>Unfortunately, Forth 94 lacks a standard VOCABULARY, and I don't
>believe one has been proposed for Forth 200x. One reason VOCABULARY is
>a good idea is that it provides a naming mechanism for wordlists, so
>that names appear in the search order listing, produced by ORDER.
>There will still be a need for two separate wordlists to be attached
>to one name: a public wordlist and a private (hidden) wordlist. AFAIK
>there is no mechanism to link two vocabularies together so that they
>could be tagged as a single module. However, if a standardized
>definition of VOCABULARY was available, and reserved at least two
>wordlists, rather than a single one, the entire module mechanism could
>be more tightly integrated with the Forth system itself.

Much preferable to such a mechanism would be a simple tag,
which could be one bit  in a flags field.
FIND (&friends) would ignore the flag, if debugging is on.

>
>Krishna

Groetjes Albert

--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

On Oct 8, 1:54=A0pm, Albert van der Horst <alb...@spenarnc.xs4all.nl>
wrote:
> In article <3c509113-57fc-4f90-9ea0-7c5ddcfaa...@h10g2000yqd.googlegroups=
..com>,
> Krishna Myneni =A0<krishna.myn...@ccreweb.org> wrote:

(1)
> >I think use of VOCABULARY would be good if standardized in Forth.
> >Unfortunately, Forth 94 lacks a standard VOCABULARY, and I don't
> >believe one has been proposed for Forth 200x. One reason VOCABULARY is
> >a good idea is that it provides a naming mechanism for wordlists, so
> >that names appear in the search order listing, produced by ORDER.

(2)
> >There will still be a need for two separate wordlists to be attached
> >to one name: a public wordlist and a private (hidden) wordlist. AFAIK
> >there is no mechanism to link two vocabularies together so that they
> >could be tagged as a single module. However, if a standardized
> >definition of VOCABULARY was available, and reserved at least two
> >wordlists, rather than a single one, the entire module mechanism could
> >be more tightly integrated with the Forth system itself.

> Much preferable to such a mechanism would be a simple tag,
> which could be one bit =A0in a flags field.
> FIND (&friends) would ignore the flag, if debugging is on.

Much preferable to which, (1) or (2)?

Much preferable to having a standard VOCABULARY so that external
source would inherit whatever facilities the local implementation
provides for vocabularies ~ such as listing the search order by
vocabulary name ...

.... or much preferable to the dual wordlist Vocabularies?

On Oct 8, 9:30=A0am, BruceMcF <agil...@netscape.net> wrote:
> On Oct 8, 9:32=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > I think the word "library" is key here. To me, a library means a
> > grouping of related modules. And, this may be why the notion of
> > storing all references to named modules into a single wordlist called
> > MODULES, bothered me. I didn't see much benefit in placing references
> > into the MODULES wordlist over placing the references into the current
> > compilation wordlist.
>
> ...
>
> As the library up the street is a collection of books on a single
> subject?
>

When you look at the files in /lib or /usr/lib on a linux system, do
you see "libraries" or "modules"?

> Some libraries would be a grouping of related modules. Others would be
> several groupings of related modules, providing a range of
> capabilities.
>
> However, an aspect of a library like FSL is that it is a "lending
> library", a means for code sharing so that capabilities can be
> *shared*.
>

Aren't all libraries intended for that purpose, ... even the real
ones?

> > Haven't thought about the need to add further definitions to an
> > existing module, elsewhere in a program... Is that a good idea?
>
> A given library may require tailoring for a specific system or
> specific task. Sometimes that will require a modified definition for
> part of the public interface.
>

In the Forth context, the present system (i.e. either DNW's root-
module.fs, your implementation, or the one I posted originally on this
thread) allows redefining the public interface(s) for a named module.
In your scheme and mine, even the private defs can be re-defined, but
the corresponding public defs which use those private defs must also
be redefined.

> For a public, external library, its often preferable to just load the
> original library and apply a patch. The module system should make it
> as straightforward as possible to apply a patch.
>
> And of course, what the source code is designed to be used for and
> what it turns out to be usable for is not always the same thing. That
> is the other reason to be skeptical about having anonymous private
> modules: even if there is some trick to apply to temporarily make them
> available, making them available by turning off the Private mode is
> only a debugging trick, its not an approach that is designed to
> provide support for code sharing.
>

If you're suggesting doing away with the anonymous module feature
altogether, that's something I could probably live with.
And even a redundant NAME-PRIVATE: word could be added, if it is
desired:

: NAME-PRIVATE: private-defs @ CREATE , DOES> @ ;

However, I would still want the ability to obtain the private wordlist
directly from the module name itself.

> And code sharing also includes two different sources that are both
> using their own selections of modules from the same library but using
> it in different ways. A system that does not make is straightforward
> to detect whether a given module has already been loaded and to take
> appropriate action is broken. It would be especially ironic if what
> makes it hard to detect and reuse code that has already been compiled
> is the fact that the application loading the library source uses the
> same module system as the library, which is to say in the case of
> encouraging public interfaces each hidden within its own named
> wordlist, the second source to be loaded having no way to know where
> the library modules were loaded by the first source to be loaded.

Hmm... you may be right. But, then, are you saying you like the idea
of LIBRARY: or not?

Krishna

On Oct 8, 6:25=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 8, 9:30=A0am, BruceMcF <agil...@netscape.net> wrote:
> > As the library up the street is a collection of books on a single
> > subject?

> When you look at the files in /lib or /usr/lib on a linux system, do
> you see "libraries" or "modules"?

When I look at /lib or /usr/lib, I see a collection of a wide range of
capabilities. As is appropros for a library directory.

> > However, an aspect of a library like FSL is that it is a "lending
> > library", a means for code sharing so that capabilities can be
> > *shared*.

> Aren't all libraries intended for that purpose, ... even the real
> ones?

Public lending libraries ~ there're private libraries where the focus
is more on collecting or preserving works.

Its a fine thing if the library modular programming system is also
usable as an application modular programming system for those who want
to use it as such ~ so long as your vision of what you would like to
see in an application modular programming system doesn't interfere
with the primary objective.

> > > Haven't thought about the need to add further definitions to an
> > > existing module, elsewhere in a program... Is that a good idea?

> > A given library may require tailoring for a specific system or
> > specific task. Sometimes that will require a modified definition for
> > part of the public interface.

> In the Forth context, the present system (i.e. either DNW's root-
> module.fs, your implementation, or the one I posted originally on this
> thread) allows redefining the public interface(s) for a named module.
> In your scheme and mine, even the private defs can be re-defined, but
> the corresponding public defs which use those private defs must also
> be redefined.

I don't reckon I've got a scheme ~ a couple of draft sketches of
modifications to dnw's and your's, but if I had to pick right now, my
preference would be a mix of SwiftForth's PACKAGES and DNW's modules.

> > For a public, external library, its often preferable to just load the
> > original library and apply a patch. The module system should make it
> > as straightforward as possible to apply a patch.
>
> > And of course, what the source code is designed to be used for and
> > what it turns out to be usable for is not always the same thing. That
> > is the other reason to be skeptical about having anonymous private
> > modules: even if there is some trick to apply to temporarily make them
> > available, making them available by turning off the Private mode is
> > only a debugging trick, its not an approach that is designed to
> > provide support for code sharing.

> If you're suggesting doing away with the anonymous module feature
> altogether, that's something I could probably live with.
> And even a redundant NAME-PRIVATE: word could be added, if it is
> desired:

> : NAME-PRIVATE: private-defs @ CREATE , DOES> @ ;

> However, I would still want the ability to obtain the private wordlist
> directly from the module name itself.

That follows directly from the private wordlist being what is named by
the module.

> > And code sharing also includes two different sources that are both
> > using their own selections of modules from the same library but using
> > it in different ways. A system that does not make is straightforward
> > to detect whether a given module has already been loaded and to take
> > appropriate action is broken. It would be especially ironic if what
> > makes it hard to detect and reuse code that has already been compiled
> > is the fact that the application loading the library source uses the
> > same module system as the library, which is to say in the case of
> > encouraging public interfaces each hidden within its own named
> > wordlist, the second source to be loaded having no way to know where
> > the library modules were loaded by the first source to be loaded.

> Hmm... you may be right. But, then, are you saying you like the idea
> of LIBRARY: or not?

You've lost me, I've been sick this week when I've had time to follow
the discussion. Let me see if I can hunt it down.

OK, yeah, LIBRARY: is fine as long as it doesn't nest ~ I'm not wed to
a single registry wordlist. MAKE-MODULE: (to make the "extend if
existing, create if not" semantics distinct) needs a place to look,
but its fine if its "the wordlist id presently in the private current-
library variable".

And the compilation wordlist when the module system is called is where
all the LIBRARY: wordlists get defined, so that LIBRARY itself can
have the "define it if missing, extend it if it exists" semantics of
PACKAGE.

As frequently, posted to save it, I'll try to test it in a couple of
systems tomorrow.

0 [IF] A simple modular programming system,
with library development extensions. [THEN]

\ First, from dnw's root-module.fs
\ a search order toolkit

[UNDEFINED] drops [IF]                 \ BMcF
   : drops ( +n -- ) 0 ?DO drop LOOP ; [THEN]

[UNDEFINED] order-drops [IF]
   : order-drops  ( +n -- ) ( o: wid_n ... wid_1 -- )
      0 ?DO previous LOOP ; [THEN]

[UNDEFINED] order-depth [IF]
   : order-depth  ( -- n )  get-order dup >r drops r> ; [THEN]

[UNDEFINED] >order [IF]
   : >order  ( wid -- order: wid )
      >r get-order r> swap 1+ set-order ; [THEN]

[UNDEFINED] order> [IF]
   : order>  ( o: wid -- ) ( -- wid )
      get-order swap >r 1- set-order r> ; [THEN]

[UNDEFINED] order@ [IF]
   : order@  ( o: wid -- ) ( -- wid )
      get-order over >r drops r> ; [THEN]

\ Implementation Note: if desired to inherit your implementation
VOCABULARY
\ Define at head of file or before calling:
\  : NAMESPACE: ( "name" -- ) VOCABULARY ; \ if you have F79 style
vocab
\  : NAMESPACE: ( "name" -- ) VOCABULARY DOES> @ >order ; \ for F83
style

[UNDEFINED] NAMESPACE: [IF]              \ BMcF, NBr
   \ define a conventional vocabulary word
   : NAMESPACE:  ( "name" -- ) wordlist CREATE ,
      DOES>  ( o: -- wid )  @ >order ; [THEN]

\ MODULES contains the modular programming words
\ in scope between MAKE-MODULE: and END-MODULE

\ MODULES is the default target for private wordlists,
\ but this can be reset

NAMESPACE: MODULES

GET-CURRENT order-depth ( current-wid0 order-depth )

MODULES DEFINITIONS

\ ************************************
\ *** MAKE-MODULE: Modules-Private ***
\ ***   by hand                    ***

NAMESPACE: Modules-Private
Modules-Private DEFINITIONS

WORDLIST DUP SET-CURRENT >order

VARIABLE prior-depth ( ... order-depth ) prior-depth !

( current-wid0 ) CONSTANT Base-wid
VARIABLE library-wid MODULES order> library-wid !
VARIABLE export-wid Base-wid export-wid !
VARIABLE public-wid Base-wid public-wid !
VARIABLE private-wid order@ private-wid !

: make-and/or-use-namespace ( container-wid "name" -- )
   >R
      >IN @ PARSE-NAME R@ SEARCH-WORDLIST 0= IF
         R@ >order DEFINITIONS
         DUP >IN ! NAMESPACE:
         DUP >IN ! ' PREVIOUS
      THEN EXECUTE DEFINITIONS DROP
   RDROP ;

PREVIOUS DEFINITIONS Modules-Private

: PRIVATE ( -- )  private-wid @ SET-CURRENT ;
: PUBLIC ( -- )  public-wid @ SET-CURRENT ;
: EXPORT ( -- ) export-wid @ SET-CURRENT ;

: SET-EXPORT ( o: wid -- wid ) order@ export-wid ! ;

\ Allow nesting of modules (not libraries)

: module> ( -- module-tag )
   prior-depth @ export-wid @ public-wid @ private-wid @
   GET-CUURENT ;

: >module ( module-tag -- )
   SET-CURRENT
   private-wid ! public-wid ! export-wid ! prior-depth ! ;

\ Restore original search order and compilation wordlist
\ Modification of the original order is an ambiguous condition

: END-MODULE  ( o: <extras> -- )
   PUBLIC order-depth prior-depth @ - order-drops ;

PUBLIC

: MAKE-MODULE: ( "name" -- )
   order-depth prior-depth !
   GET-CURRENT DUP public-wid ! DUP export-wid !
   order@ <> IF GET-CURRENT >order THEN
   MODULES
   library-wid @ make-and/or-use-namespace
   order@ private-wid ! ;

: LIBRARY: ( "name" -- ) Base-wid make-and/or-use-namespace ;

In article <1cb26beb-1093-469a-bd76-5e993ad9c6e3@h10g2000yqd.googlegroups.com>,
BruceMcF  <agila61@netscape.net> wrote:
>On Oct 8, 1:54=A0pm, Albert van der Horst <alb...@spenarnc.xs4all.nl>
>wrote:
>> In article <3c509113-57fc-4f90-9ea0-7c5ddcfaa...@h10g2000yqd.googlegroups=
>.com>,
>> Krishna Myneni =A0<krishna.myn...@ccreweb.org> wrote:
>
>(1)
>> >I think use of VOCABULARY would be good if standardized in Forth.
>> >Unfortunately, Forth 94 lacks a standard VOCABULARY, and I don't
>> >believe one has been proposed for Forth 200x. One reason VOCABULARY is
>> >a good idea is that it provides a naming mechanism for wordlists, so
>> >that names appear in the search order listing, produced by ORDER.
>
>(2)
>> >There will still be a need for two separate wordlists to be attached
>> >to one name: a public wordlist and a private (hidden) wordlist. AFAIK
>> >there is no mechanism to link two vocabularies together so that they
>> >could be tagged as a single module. However, if a standardized
>> >definition of VOCABULARY was available, and reserved at least two
>> >wordlists, rather than a single one, the entire module mechanism could
>> >be more tightly integrated with the Forth system itself.
>
>> Much preferable to such a mechanism would be a simple tag,
>> which could be one bit =A0in a flags field.
>> FIND (&friends) would ignore the flag, if debugging is on.
>
>Much preferable to which, (1) or (2)?
>
>Much preferable to having a standard VOCABULARY so that external
>source would inherit whatever facilities the local implementation
>provides for vocabularies ~ such as listing the search order by
>vocabulary name ...
>
>... or much preferable to the dual wordlist Vocabularies?

I meant dual wordlist vocabularies.

Groetjes Albert


--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

BruceMcF wrote:
> \  : NAMESPACE: ( "name" -- ) VOCABULARY ; \ if you have F79 style
> vocab
> \  : NAMESPACE: ( "name" -- ) VOCABULARY DOES> @ >order ; \ for F83
> style

I'm not up on this aspect of Forth history, but I *think* you have
that backwards.  Forth-83 seems to have a search order, so I'm
guessing it was Forth-79 which had the upward-linked tree thingy.

--Josh

On Oct 8, 6:58=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 8, 6:25=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > On Oct 8, 9:30=A0am, BruceMcF <agil...@netscape.net> wrote:
> > > As the library up the street is a collection of books on a single
> > > subject?
> > When you look at the files in /lib or /usr/lib on a linux system, do
> > you see "libraries" or "modules"?
>
> When I look at /lib or /usr/lib, I see a collection of a wide range of
> capabilities. As is appropros for a library directory.
>
> > > However, an aspect of a library like FSL is that it is a "lending
> > > library", a means for code sharing so that capabilities can be
> > > *shared*.
> > Aren't all libraries intended for that purpose, ... even the real
> > ones?
>
> Public lending libraries ~ there're private libraries where the focus
> is more on collecting or preserving works.
>
> Its a fine thing if the library modular programming system is also
> usable as an application modular programming system for those who want
> to use it as such ~ so long as your vision of what you would like to
> see in an application modular programming system doesn't interfere
> with the primary objective.
>
> > > > Haven't thought about the need to add further definitions to an
> > > > existing module, elsewhere in a program... Is that a good idea?
> > > A given library may require tailoring for a specific system or
> > > specific task. Sometimes that will require a modified definition for
> > > part of the public interface.
> > In the Forth context, the present system (i.e. either DNW's root-
> > module.fs, your implementation, or the one I posted originally on this
> > thread) allows redefining the public interface(s) for a named module.
> > In your scheme and mine, even the private defs can be re-defined, but
> > the corresponding public defs which use those private defs must also
> > be redefined.
>
> I don't reckon I've got a scheme ~ a couple of draft sketches of
> modifications to dnw's and your's, but if I had to pick right now, my
> preference would be a mix of SwiftForth's PACKAGES and DNW's modules.
>
>

After the discussions here, and my own attempt to write a modules
interface, I've come to have a much greater appreciation for David's
root-module.fs.

>
> > > For a public, external library, its often preferable to just load the
> > > original library and apply a patch. The module system should make it
> > > as straightforward as possible to apply a patch.
>
> > > And of course, what the source code is designed to be used for and
> > > what it turns out to be usable for is not always the same thing. That
> > > is the other reason to be skeptical about having anonymous private
> > > modules: even if there is some trick to apply to temporarily make the=
m
> > > available, making them available by turning off the Private mode is
> > > only a debugging trick, its not an approach that is designed to
> > > provide support for code sharing.
> > If you're suggesting doing away with the anonymous module feature
> > altogether, that's something I could probably live with.
> > And even a redundant NAME-PRIVATE: word could be added, if it is
> > desired:
> > : NAME-PRIVATE: private-defs @ CREATE , DOES> @ ;
> > However, I would still want the ability to obtain the private wordlist
> > directly from the module name itself.
>
> That follows directly from the private wordlist being what is named by
> the module.
>

Just so I didn't misunderstand your statement above. I'm under the
impression that everyone (in this discussion) agrees that executing
the name of a module pushes its *public* wordlist into the search
order. The question is how to push the private wordlist onto the
search order, which would only need to be done, presumably, for
debugging a module.

> > > And code sharing also includes two different sources that are both
> > > using their own selections of modules from the same library but using
> > > it in different ways. A system that does not make is straightforward
> > > to detect whether a given module has already been loaded and to take
> > > appropriate action is broken. It would be especially ironic if what
> > > makes it hard to detect and reuse code that has already been compiled
> > > is the fact that the application loading the library source uses the
> > > same module system as the library, which is to say in the case of
> > > encouraging public interfaces each hidden within its own named
> > > wordlist, the second source to be loaded having no way to know where
> > > the library modules were loaded by the first source to be loaded.
> > Hmm... you may be right. But, then, are you saying you like the idea
> > of LIBRARY: or not?
>
> You've lost me, I've been sick this week when I've had time to follow
> the discussion. Let me see if I can hunt it down.
>
> OK, yeah, LIBRARY: is fine as long as it doesn't nest ~ I'm not wed to
> a single registry wordlist. MAKE-MODULE: (to make the "extend if
> existing, create if not" semantics distinct) needs a place to look,
> but its fine if its "the wordlist id presently in the private current-
> library variable".
>
> And the compilation wordlist when the module system is called is where
> all the LIBRARY: wordlists get defined, so that LIBRARY itself can
> have the "define it if missing, extend it if it exists" semantics of
> PACKAGE.

What do you mean by "when the module system is called". Do you mean
when the modular programming framework, e.g. root-module.fs, is
loaded?

Krishna

On Oct 8, 12:54=A0pm, Albert van der Horst <alb...@spenarnc.xs4all.nl>
wrote:
> In article <3c509113-57fc-4f90-9ea0-7c5ddcfaa...@h10g2000yqd.googlegroups=
..com>,
> Krishna Myneni =A0<krishna.myn...@ccreweb.org> wrote:
>
>
>
> >I think use of VOCABULARY would be good if standardized in Forth.
> >Unfortunately, Forth 94 lacks a standard VOCABULARY, and I don't
> >believe one has been proposed for Forth 200x. One reason VOCABULARY is
> >a good idea is that it provides a naming mechanism for wordlists, so
> >that names appear in the search order listing, produced by ORDER.
> >There will still be a need for two separate wordlists to be attached
> >to one name: a public wordlist and a private (hidden) wordlist. AFAIK
> >there is no mechanism to link two vocabularies together so that they
> >could be tagged as a single module. However, if a standardized
> >definition of VOCABULARY was available, and reserved at least two
> >wordlists, rather than a single one, the entire module mechanism could
> >be more tightly integrated with the Forth system itself.
>
> Much preferable to such a mechanism would be a simple tag,
> which could be one bit =A0in a flags field.
> FIND (&friends) would ignore the flag, if debugging is on.
>
>
>
> >Krishna
>
> Groetjes Albert
>
> --
> --
> Albert van der Horst, UTRECHT,THE NETHERLANDS
> Economic growth -- being exponential -- ultimately falters.
> albert@spe&ar&c.xs4all.nl &=3Dnhttp://home.hccnet.nl/a.w.m.van.der.horst

I don't see how to use your suggestion to build a module having both a
public and private wordlist attached to it. Could you sketch an
example?

I'm just speculating on what (minimal) structural changes in Forth
might support a modules framework to be implemented more simply.
Similarly, one could also consider minimal changes which would provide
support for OO implementations in Forth. These changes would not
dictate that actual implementations, but would provide some needed
infrastructure so that such frameworks can be implemented more easily.

Krishna

On 09.10.2011 15:16, Krishna Myneni wrote:
> I'm just speculating on what (minimal) structural changes in Forth
> might support a modules framework to be implemented more simply.
> Similarly, one could also consider minimal changes which would provide
> support for OO implementations in Forth. These changes would not
> dictate that actual implementations, but would provide some needed
> infrastructure so that such frameworks can be implemented more easily.
>
> Krishna

Whether changes are minimal or not depends on the system.
But on fact remains:
Namespaces are for code organization at compile-time, wordlists for 
search organization at run-time. Both are not just different shoes, but 
fundamentally different things.

IMO I would organize a more elaborated Forth namespace in a tree-like 
directory structure. The Forths I know search only linear wordlists. The 
search order is also linear and mandates the sequence. This is the 
classic search method, which works also in tree-like namespaces (by 
searching always just one branch = wordlist).

Another search method in the same unmodified(!) namespace would descend 
down to sub-branches. This could provide the infrastructure for OO class 
hierarchies.

OTOH in my world it would be just a leisure project. I don't need it in 
reality. So what the heck...
Why making Forth more complex? Keep it simple!

Andreas

On Oct 9, 8:24=A0am, Josh Grams <j...@qualdan.com> wrote:
> BruceMcF wrote:
> > \ =A0: NAMESPACE: ( "name" -- ) VOCABULARY ; \ if you have F79 style
> > vocab
> > \ =A0: NAMESPACE: ( "name" -- ) VOCABULARY DOES> @ >order ; \ for F83
> > style
>
> I'm not up on this aspect of Forth history, but I *think* you have
> that backwards. =A0Forth-83 seems to have a search order, so I'm
> guessing it was Forth-79 which had the upward-linked tree thingy.
>
> --Josh

AFAIR, it was fig-Forth that had the tree, where the vocabulary
chained in at the point where it had been created. The core Forth 79
was just a vocabulary pointer in a CONTEXT variable, and if the
CONTEXT was not the Forth wordlist, then Forth was searched when the
CONTEXT wordlist was exhausted. DEFINITIONS copied CONTEXT to CURRENT.

There was no explicit search order stack in the informal Forth 79
standard, but Forth 79 systems that allowed multiple simultaneous
vocabularies did it with either a linked list or a search order stack.
Then if you had an block editor vocabulary and an assembler
vocabulary, you could have them both available, rather than swapping
back and forth, which is convenient when editing and testing blocks
containing assembly language.

A common practice was that those vocabularies pushed themselves into
the front of the search order. That wasn't a Forth 79 standard, and it
wasn't the only practice, but AFAICR it was a common one. Then if you
always start with the Forth wordlist, you just list the vocabularies
you want:
   FORTH ED ASM02
.... with FORTH the target for definitions unless explicitly overridden
by DEFINITIONS.

Of course, that doesn't precisely emulate how a Forth79 system without
an explicit search *worked* when you are using multiple vocabularies,
which would have a sequence of

FORTH ED ... ASM02 ... ED ... ASM02 ...

Forth 83 had an explicit search order, controlled by the ONLY FORTH
ALSO VOCABULARY DEFINITIONS system, and an explicit affect of a
VOCABULARY on the search order of replacing what was there before ~ an
extension of the basic Forth79 system, but in conflict which what
several Forth79 systems that already had a search order commonly did.

On Oct 9, 9:09=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:

> Just so I didn't misunderstand your statement above. I'm under the
> impression that everyone (in this discussion) agrees that executing
> the name of a module pushes its *public* wordlist into the search
> order.

No, after digesting some of the objections to modular programming at
all, I prefer a library module system to follow the SwiftForth
approach where the PUBLIC wordlist *is* the compilation wordlist when
the PACKAGE is called, and the PRIVATE wordlist is what is named by
the PACKAGE.

 The question is how to push the private wordlist onto the
> search order, which would only need to be done, presumably, for
> debugging a module.

*Or* if there are useful facilities contained in the private wordlist,
and you want to make use of them.

For programming and system space efficiency, the second is every bit
as important as the first.

On Oct 9, 7:45=A0am, Albert van der Horst <alb...@spenarnc.xs4all.nl>
wrote:

> BruceMcF =A0<agil...@netscape.net> wrote:
> >Much preferable to which, (1) or (2)?

> >Much preferable to having a standard VOCABULARY so that external
> >source would inherit whatever facilities the local implementation
> >provides for vocabularies ~ such as listing the search order by
> >vocabulary name ...

> >... or much preferable to the dual wordlist Vocabularies?

> I meant dual wordlist vocabularies.

Yes, I think that dual wordlist vocabularies are barking up the wrong
tree.

On Oct 9, 1:21=A0pm, "A. K." <a...@nospam.org> wrote:
> Namespaces are for code organization at compile-time, wordlists for
> search organization at run-time. Both are not just different shoes, but
> fundamentally different things.

Compile time *is* the run time of the compiler words. If run time and
compile time were turned into fundamentally different things in Forth,
then Forth would be turned into a fundamentally different animal in
the process.

On Oct 9, 12:21=A0pm, "A. K." <a...@nospam.org> wrote:
> On 09.10.2011 15:16, Krishna Myneni wrote:
>
> > I'm just speculating on what (minimal) structural changes in Forth
> > might support a modules framework to be implemented more simply.
> > Similarly, one could also consider minimal changes which would provide
> > support for OO implementations in Forth. These changes would not
> > dictate that actual implementations, but would provide some needed
> > infrastructure so that such frameworks can be implemented more easily.
>
> > Krishna
>
> Whether changes are minimal or not depends on the system.
> But on fact remains:
> Namespaces are for code organization at compile-time, wordlists for
> search organization at run-time. Both are not just different shoes, but
> fundamentally different things.
>

No, not really. Forth is interactive. That makes the distinction less
meaningful. Especially when we are doing interactive debugging from
the Forth environment.

> IMO I would organize a more elaborated Forth namespace in a tree-like
> directory structure. The Forths I know search only linear wordlists. The
> search order is also linear and mandates the sequence. This is the
> classic search method, which works also in tree-like namespaces (by
> searching always just one branch =3D wordlist).
>

What if a name appears on more than one branch? Which one do you pick?
The one with the least depth?

> Another search method in the same unmodified(!) namespace would descend
> down to sub-branches. This could provide the infrastructure for OO class
> hierarchies.
>
> OTOH in my world it would be just a leisure project. I don't need it in
> reality. So what the heck...
> Why making Forth more complex? Keep it simple!
>
> Andreas

Minimum required level of complexity to deal with a real programming
need, I think, is the key. The FSL already demonstrates (to me) that
its simple modularity method is inadequate in a larger setting.

Krishna

On Oct 9, 12:38=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 9, 9:09=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > Just so I didn't misunderstand your statement above. I'm under the
> > impression that everyone (in this discussion) agrees that executing
> > the name of a module pushes its *public* wordlist into the search
> > order.
>
> No, after digesting some of the objections to modular programming at
> all, I prefer a library module system to follow the SwiftForth
> approach where the PUBLIC wordlist *is* the compilation wordlist when
> the PACKAGE is called, and the PRIVATE wordlist is what is named by
> the PACKAGE.
> ...

Ah, so the only point of agreement I thought we had, really isn't.
Offhand I don't agree that the module name should provide the private
wordlist. What you describe is an anonymous module with a named
private wordlist. If NAME-PRIVATE: is added to my existing module.fs,
it would provide the same functionality.

Krishna

VOCABULARY was too modular in it's design, as shown by the ASM and ED neede=
d at the same time example. The original solution to this involved hacking =
the error jump cell in FIND and then go through a DO LOOP doing a CONTEXT s=
et and switch.

The word CONTEXT was also hacked to populate this list of alternates automa=
tically. This required the list to be finite in size, or an extra cell addi=
ng to VOCABULARY words to make a priority chain. It required one extra word=
 to clean the alternate list, and reset CONTEXT to FORTH. Why this extra wo=
rd was extended to accept a potentially uncertain vocabulary name (multiple=
 defs) is anybodies guess...

Usually context in human terms is provided by a sense of history, and the i=
dea of forgetting. It maybe never occurred to some that FORTH is searched b=
y chain for any CONTEXT and so, serves to obtain a base. And all because th=
e not found error disappeared?

ONLY and ALSO are to my mind, unnecessary, and the word PEDANT for flushing=
 the alternate list should be fine.

Structure is an important thing, and much was eliminated by including CREAT=
E DOES> and IMMEDIATE so as to limit forward compilation. But knowing of th=
e error at compile time is perhaps better than a runtime "word not yet defi=
ned".

What should modules achieve.

Cheers Jacko

On Oct 9, 3:50=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> Ah, so the only point of agreement I thought we had, really isn't.

I'm not sure where you got the idea I wanted to have to include a
module file, then define the public interface out into the actual
target namespace with a bunch of
   : foo foo ;
   : bar bar ;
   ...
.... definitions. The public words in that included file *ought* to
show up in the namespace that receives definitions when I include it.

> Offhand I don't agree that the module name should provide the private
> wordlist. What you describe is an anonymous module with a named
> private wordlist.

A fairly misleading description ~ in the original anonymous module,
the private was necessarily anonymous, but as I noted, whether the
public is anonymous or not is left up to the includer.

More descriptive is inherited public, named private. Forth Inc. has
found that that suffices where public / private definitions are
required, and it avoids multiple complexities that might be fun to
solve, but impose unnecessary restrictions on the users.

> If NAME-PRIVATE: is added to my existing module.fs,
> it would provide the same functionality.

Except NAME-PRIVATE: does *not* provide the same functionality *in
practice*, since it only works without editing for those modules where
the *author* can envision some purpose for later access to the private
words: it doesn't work for any private namespace where the *user*
comes across a purpose at hand.

The library module author ensuring that the only word conflicts that
*have* to be accommodated is the public interface ~ that's a feature.
The library module author denying me the use of some unanticipated use
of his code ~ that's a bug.

And while having the public interface for each module sealed off in a
distinct named wordlist is inconvenient for most users, the flipside
is the ability to detect whether the module has been loaded at all ~
so inheriting the public namespace and naming the private namespace
kills three birds with one stone ~ maximum compatibility with the host
system approach to namespace management, being open to unanticipated
re-use of existing code, and a name at the module level which, if
distinct names are given to distinct versions of a module, can be used
to detect whether the required version of the module is available.

: UNIT: CONTEXT @ VOCABULARY DEFINITIONS ; 
: EXPORT CURRENT @ DUP @ CURRENT ! FIND EXECUTE CURRENT ! ; 
: ;UNIT CURRENT @ @ CURRENT ! CONTEXT ! ;

Is about my best in a single context environment. EXPORT works as a prefix on : CONSTANT and other definers. Needs checking.

Cheers Jacko

: UNIT: CONTEXT @ CURRENT @ VOCABULARY DEFINITIONS ; 
: EXPORT CURRENT @ OVER CURRENT ! FIND EXECUTE CURRENT ! ; 
: ;UNIT CURRENT ! CONTEXT ! ;

Is more like it. As CURRENT @ @ is a pointer to the second word in the current search chain. Dooough

Cheers Jacko

On Oct 9, 8:29=A0pm, jacko <jackokr...@gmail.com> wrote:
> VOCABULARY was too modular in it's design, as shown by the ASM and ED nee=
ded at the same time example.

I don't think "modular" is the word. But one supplementary vocabulary
at a time was limiting, and people addressed that limit, given that
they had the tools at their disposal to do so.

> The original solution to this involved hacking the error jump cell in FIN=
D and then go through a DO LOOP doing a CONTEXT set and switch.

> The word CONTEXT was also hacked to populate this list of alternates auto=
matically. This required the list to be finite in size, or an extra cell ad=
ding to VOCABULARY words to make a priority chain.

Or faking out FIND with dummy "nul" entries embedded in the body of
the word defined by VOCABULARY, in which case the vocabulary when it
executes simply copies the current CONTEXT into the <nextword> field
of its dummy "next vocabulary" entry, before it copies its own address
into CONTEXT.

This had a certain appeal as an add on, since it only changed
VOCABULARY, and did not require FIND to be modified to cope with
multiple <cell-next>=3D0 terminated wordlists.

But in any event, that approach of modifying VOCABULARY is the minimal
extension to multiple supplementary vocabularies, in the sense that it
uses the same three word system: FORTH DEFINITIONS and VOCABULARY.

> It required one extra word to clean the alternate list, and reset CONTEXT=
 to FORTH. Why this extra word was extended to accept a potentially uncerta=
in vocabulary name (multiple defs) is anybodies guess...

Is this referring to the way that Forth79 FORTH worked, both before
and after extending to multiple supplementary vocabularies as above,
or to Forth83's ONLY?

I think the hobbyist Forth83 standard had a specific approach in mind,
but when Forth94 came around, there were multiple actual
implementations of the search order stack to support, so the Forth94
standard had to be vague on whether the normal start up state of the
search order stack is a base wordlist plus the forth wordlist or the
forth wordlist as the base wordlist.

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> On Oct 9, 12:38?pm, BruceMcF <agil...@netscape.net> wrote:
>> On Oct 9, 9:09?am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>>
>> > Just so I didn't misunderstand your statement above. I'm under
>> > the impression that everyone (in this discussion) agrees that
>> > executing the name of a module pushes its *public* wordlist into
>> > the search order.
>>
>> No, after digesting some of the objections to modular programming at
>> all, I prefer a library module system to follow the SwiftForth
>> approach where the PUBLIC wordlist *is* the compilation wordlist when
>> the PACKAGE is called, and the PRIVATE wordlist is what is named by
>> the PACKAGE.
>> ...
> 
> Ah, so the only point of agreement I thought we had, really isn't.
> Offhand I don't agree that the module name should provide the private
> wordlist.

Surely this should be up to the includer of a library.  If you really
want a library's public definitions to be in a separate wordlist you
can do, e.g.

vocabulary trigonometry
also trigonometry definitions
include trigonometry.f

Andrew.

On Oct 10, 4:41=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > On Oct 9, 12:38?pm, BruceMcF <agil...@netscape.net> wrote:
> >> On Oct 9, 9:09?am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> >> > Just so I didn't misunderstand your statement above. I'm under
> >> > the impression that everyone (in this discussion) agrees that
> >> > executing the name of a module pushes its *public* wordlist into
> >> > the search order.
>
> >> No, after digesting some of the objections to modular programming at
> >> all, I prefer a library module system to follow the SwiftForth
> >> approach where the PUBLIC wordlist *is* the compilation wordlist when
> >> the PACKAGE is called, and the PRIVATE wordlist is what is named by
> >> the PACKAGE.
> >> ...
>
> > Ah, so the only point of agreement I thought we had, really isn't.
> > Offhand I don't agree that the module name should provide the private
> > wordlist.
>
> Surely this should be up to the includer of a library. =A0If you really
> want a library's public definitions to be in a separate wordlist you
> can do, e.g.
>
> vocabulary trigonometry
> also trigonometry definitions
> include trigonometry.f

Or, equally,

wordlist constant trigonometry
trigonometry use-wordlist
include trigonometry.fs

A library putting its public definitions into a wordlist defined by
the library also implies exporting either a F83 vocabulary, a self-
adding vocabulary, or a wordlist constant to a host that, as should be
obvious, may be set up to use one of the other two approaches.

On Oct 9, 7:29=A0pm, jacko <jackokr...@gmail.com> wrote:

> ...
> What should modules achieve.
> ...

A very good question, and one to which there will probably be no
uniform reply. However, I can give you my take on what programming
facility should be provided by modules. Also, given that there are
several existing implementations (FSL, Neal Bridges' modules.x, David
Williams' root-modules.fs, Swift Forth's package facility, etc.),
there is a need for some form of enhanced ordering/control of
compilation wordlist and search order in Forth.

Here are my expectations for a modules facility in Forth.

1) It should allow a group of definitions related to a single task to
be organized into a distinctly named entity, called the module.

2) The definitions within a module should fall into two groups: public
and private. The public definitions constitute the interface to code
external to the module, while the private definitions support the
public definitions, but are not needed, and therefore not visible, to
external code. Private definitions are isolated from being referenced,
and therefore, from being modified, by external code. An exception
mechanism allows private definitions to be made visible for debugging
from the interpreter, and also for rare cases when direct access to
private definitions by external code is more convenient than a public
interface. In general, however, public definitions should provide all
of the necessary interface to interact with the module.

3) The module can be addressable by name in order to explicitly bring
its definitions into the search order, for use by subsequent modules
or other external code which make use of its public interfaces.

4) The modular programming facility should make it easy to load
modules into the system without adverse side effects to the
compilation wordlist and search order. That is, it provides some
guarantees on the end state of the system, with respect to the
compilation wordlist and search order, after a module has been loaded,
thereby relieving the programmer of having to worry about resetting
the state of the system after each load.


Krishna

On 06/10/2011 00:03, Krishna Myneni wrote:
> Below is my bare-bones sketch of an interface for modular programming
> in Forth, adapted from the code of Bruce McFarling, David N. Williams,
> and Neal Bridges. I think it provides the essential features necessary
> for modular programming, with the exceptions of a wordlist to collect
> the module references, and Bruce's HELPER words. The code is very
> preliminary, and my testing has been only cursory, using the test
> examples given below. Feel free to offer criticisms and suggestions.
>
> Krishna
> --
> \  First, the demo/test examples
>
> \ module-test.fs
> \
> \
> include module.fs
>
> \ Ex. 1: anonymous module
>
> begin-module
>
> 15 constant pr1
>
> Public:
>
> : pu1 ." Anonymous public word " pr1 . ;
>
> end-module
>
>
> \ Ex. 2: named module Foo
> module: Foo
> begin-module
>
> 5 constant pr2
>
> Public:
>
> 20 constant pu2a
>
> : pu2 ." Foo public word " pr2 . ;
>
> end-module
>
>
> \ Ex. 3: Anonymous module using Foo
>
> begin-module
> Foo           \ depends on Foo
>
> Public:
>
> 10 constant pu3a
>
> : pu3 ." Anonymous public word calls " pu2 ;
>
> end-module
>
> \ Ex. 4: Named module Bar using Foo and anonymous
>
> module: Bar
> begin-module
> Foo
>
> pu2a pu3a + constant pr4
>
> Public:
>
> : pu4 ." Bar public word calls " pu3  ;
>
> end-module
>
> \ Ex. 5: Access private member of Bar
>
> ' Bar>private>order
> cr .( Private member of Bar: ) pr4 . cr
>
> previous
>
>
> -------------------------------------------
> \ Now, the interface
>
> (      Title:  Modular programming in Forth
>          File:  module.fs
>       Revised:  October 5, 2011
>
> Description:
> -----------
>
> This library provides a facility for modular programming in
> Forth. It supports both named and anonymous modules, and allows
> for private definitions in a module to be made accessible.
> The library uses design features from several sources:
> David N. Williams' root-module.fs, Bruce McFarling's suggested
> module facility for the Forth Scientific Library [FSL], the
> original module facility in the FSL's fsl-util auxiliary files, and
> Neal Bridges' anonymous modules code.
>
>
> The layout of a module is determined by the following words:
>
>    MODULE:  BEGIN-MODULE  PUBLIC:  PRIVATE:  END-MODULE
>
> Here, BEGIN-MODULE is where the module begins, and END-MODULE
> is where it ends.  The code area between the two is "inside" the
> module. The words PUBLIC: and PRIVATE: control the visibility
> of the module code to external code and to the user. The use of
> MODULE: , to provide a named reference to the module, is
> optional.
>
>
> Terminology:
> -----------
>
> 1. The term "anonymous module" is a module which is
>     unnamed.
>
>
> Design elements:
> ---------------
>
> 1. A module may or may not be named:
>
>     a] For an unnamed [anonymous] module, the public definitions
>        are compiled into the current compilation wordlist.
>
>     b] For a named module, the public definitions are compiled
>        into a new wordlist. The module name may be used
>        to add the public wordlist to the search order, for use
>        by other modules or external code.
>
>     c] For both a named and anonymous module, a new private
>        wordlist is created. For named modules, the private
>        wordlist id can be retrieved using>PRIVATE on the
>        module xt. The private wordlist of an anonymous module
>        is not accessible outside the module.
>
> 2. The module layout is not nestable.
>
> 3. The search order depth and compilation wordlist just before
>     BEGIN-MODULE are restored by END-MODULE.
>
> 4. PRIVATE: and PUBLIC: should occur only inside a module, and may
>     be invoked any number of times there, in any order.  Definitions
>     inside a module usually go into its public or private
>     wordlist.
>
> 5. The part of the search order present just before the
>     beginning of a module may not be changed inside the module.
>
>     [This could be made safer by actually saving and restoring
>     the initial search order.  This implementation does not do
>     that.]
> )
>
>
> \ *** GENERAL USE
>
> [UNDEFINED] drops [IF]                 \ BMcF
> : drops ( +n -- ) 0 ?DO drop LOOP ; [THEN]
>
> [UNDEFINED] order-drops [IF]
> : order-drops  ( +n -- ) ( o: wid_n ... wid_1 -- )
>    0 ?DO previous LOOP ; [THEN]
>
> [UNDEFINED] order-depth [IF]
> : order-depth  ( -- n )  get-order dup>r drops r>  ; [THEN]
>
> [UNDEFINED]>order [IF]      \ same as BMcF's ADD-WORDLIST
> :>order  ( wid -- order: wid )
>    >r get-order r>  swap 1+ set-order ; [THEN]
>
> [UNDEFINED] order>  [IF]  \ not used in this module
> : order>   ( o: wid -- ) ( -- wid )
>    get-order swap>r 1- set-order r>  ; [THEN]
>
> [UNDEFINED] order@ [IF]  \ not used in this module
> : order@  ( o: wid -- ) ( -- wid )
>    get-order over>r drops r>  ; [THEN]
>
>
> VARIABLE initial-defs         get-current initial-defs !
> VARIABLE initial-order-depth
> VARIABLE private-defs         get-current private-defs !
> VARIABLE public-defs          initial-defs @ public-defs !
> VARIABLE named-module         false named-module !
>
> \ Named module
>
> [UNDEFINED] MODULE: [IF]
> \ define an extended vocabulary word for a named module
> : MODULE:  ( "name" -- )
>      get-current  initial-defs !
>      true         named-module !
>      \ create the private and public wordlists
>      wordlist wordlist 2dup CREATE , ,
>      set-current  private-defs !
>    DOES>   ( o: -- wid )  @>order  \ add public wordlist to search
> order
> ;
> [THEN]
>
> \ Return private wordlist id from the xt of a named module
> :>PRIVATE ( xtmodule -- wid )>body cell+ @ ;
>
> \ *** PUBLIC DEFINITIONS
>
> \ public-defs @ set-current  \ same as initial-defs
>
> \ Defns of PRIVATE: and PUBLIC: are equivalent to BMcF's
> : PRIVATE:  ( -- )  private-defs @ set-current ;
> : PUBLIC:   ( -- )  public-defs  @ set-current ;
>
> \ BEGIN-MODULE and END-MODULE are not to be nested.
>
> : BEGIN-MODULE  ( -- ) ( o: -- public private )
> (
> Save information that allows END-MODULE to restore the initial
> search order and compilation wordlist, assuming the initial
> order has remained unchanged.  Define "name" as a vocabulary
> word belonging to the MODULES wordlist, and add its wordlist,
> "public", to the search order.  Create an anonymous wordlist,
> "private", add it to the search order, and make it the
> compilation wordlist.  Store the public and private wids for use
> by PUBLIC: and PRIVATE:.
> )
>    order-depth  initial-order-depth !
>    get-current  dup>order public-defs !
>    named-module @ if
>      private-defs @
>    else
>      public-defs @ initial-defs !
>      wordlist dup private-defs !
>    then
>    dup>order
>    set-current    \ default section is private
> ;
>
> : END-MODULE  ( o:<extras>  -- )
> (
> Restore the initial search order depth and compilation wordlist.  An
> ambiguous condition exists if the initial search order has been
> clobbered.
> )
>    initial-defs @ set-current
>    order-depth initial-order-depth @ - order-drops
>    false  named-module ! ;
>
>
> \ --------------------------------------------
>
> \ The examples below assume that module.fs is loaded.
>
> \ EXAMPLE 1: Layout for anonymous module
> 0 [IF]
>
> BEGIN-MODULE
> \ Add named modules and wordlists to the search order on which
> \ this module depends.  They will be removed from the top of the
> \ search order By END-MODULE.  Keeping the list of modules on
> \ which it depends inside a module more clearly documents its
> \ dependencies, whereas adding modules before BEGIN-MODULE:
> \ makes it easier to introduce inadvertent, unwanted
> \ dependencies.
>
> <mod_1>  ...<mod_n>
>
> \ Initial definitions are private, and placed in a hidden
> \   wordlist accessible only to this module.
>
> <initial private definitions>
>
> PUBLIC:
> \ public definitions go into the current compilation wordlist
>
> \ any number of subsequent PRIVATE: and PUBLIC: sections follow,
> \ in any order.
>
> END-MODULE
> [THEN]  \ END EXAMPLE 1 LAYOUT
> \ --------------------------------------------
>
> \ EXAMPLE 2: Layout for a named module
> 0 [IF]
>
> \ Make a vocabulary for the public definitions of a
> \ module named Foo.
>
> Module: Foo
>
> BEGIN-MODULE
>
> \ Add named modules and wordlists to the search order on which
> \ this module depends.
> <mod_1>  ...<mod_n>
>
> PUBLIC:
> \ public definitions go into the wordlist, Foo.
>
> \ any number of subsequent private and public sections follow,
> \ in any order.
>
> END-MODULE
> [THEN]  \ END EXAMPLE 2 LAYOUT
> \ --------------------------------------------
>
> \ To access the private definitions of Foo from another
> \ module, retrieve the private wordlist of Foo, and add it
> \ to the search order, e.g.
> \
> \   ' Foo>private>order
>
> --------------------------------
>

I've been following the discussion with some interest and, while I have 
sort of lost track of where different people stand on the topic, it 
seems to me that a consensus is not emerging. I am increasingly feeling 
that it might be better to split responsibilities for modularisation of 
code between the modules and the user.

For example if there were 4 words available for use by a module:

BEGIN-MODULE which saves the compilation wordlist and top wordlist in 
the search order as the public and private wordlists respectively, 
setting the private wordlist as the compilation wordlist.

PRIVATE and PUBLIC setting their wordlists as the compilation wordlist.

END-MODULE restores the original compilation wordlist

Then it would be up to the user to decide whether to name a module or 
not, whether to create a new public wordlist or not, whether to have a 
private wordlist (use the public one for both if a flat namespace is 
wanted) and so on.

The user sets the strategy, the module does the mechanics.

-- 
Gerry

Using a vocabulary word also had to remove itself from the search order bef=
ore placing it at the front in order to prevent a circular search order, du=
e to only having one link pointer. This of course had effects on restoring =
CURRENT or CONTEXT, as the vocabulary could have been excised from the sear=
ch order.

This minor bug is livable in most situations, and probably preferred to a f=
ull search order mash up. In terms of modules, this bug a.k.a feature would=
 remove the private definitions vocabulary from the search order, and so be=
 a kind of Zen, if it were not for the fact the vocabulary would by means o=
f creation be fore most in the search order. But it would remove any nested=
 module privates from the search order too. It could perhaps remove FORTH o=
r any other referenced within a module vocabulary, which would be bad, bad,=
 bad!

Hence the CONTEXT hack. Better an aphasic forth than a vegetable brick! Hen=
ce the always check FORTH last idea. With the modules/units idea the multip=
le vocabs on the search order disappears though. So why need a search order=
? One CONTEXT is enough. Premature optimization is the root of all complexi=
ty in implementation of structured programming words ;P

Cheers Jacko=20

On Oct 10, 10:04=A0am, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
wrote:
> On 06/10/2011 00:03, Krishna Myneni wrote:
>
>
>
> > Below is my bare-bones sketch of an interface for modular programming
> > in Forth, adapted from the code of Bruce McFarling, David N. Williams,
> > and Neal Bridges. I think it provides the essential features necessary
> > for modular programming, with the exceptions of a wordlist to collect
> > the module references, and Bruce's HELPER words. The code is very
> > preliminary, and my testing has been only cursory, using the test
> > examples given below. Feel free to offer criticisms and suggestions.
>
> > Krishna
> > --
> > \ =A0First, the demo/test examples
>
> > \ module-test.fs
> > \
> > \
> > include module.fs
>
> > \ Ex. 1: anonymous module
>
> > begin-module
>
> > 15 constant pr1
>
> > Public:
>
> > : pu1 ." Anonymous public word " pr1 . ;
>
> > end-module
>
> > \ Ex. 2: named module Foo
> > module: Foo
> > begin-module
>
> > 5 constant pr2
>
> > Public:
>
> > 20 constant pu2a
>
> > : pu2 ." Foo public word " pr2 . ;
>
> > end-module
>
> > \ Ex. 3: Anonymous module using Foo
>
> > begin-module
> > Foo =A0 =A0 =A0 =A0 =A0 \ depends on Foo
>
> > Public:
>
> > 10 constant pu3a
>
> > : pu3 ." Anonymous public word calls " pu2 ;
>
> > end-module
>
> > \ Ex. 4: Named module Bar using Foo and anonymous
>
> > module: Bar
> > begin-module
> > Foo
>
> > pu2a pu3a + constant pr4
>
> > Public:
>
> > : pu4 ." Bar public word calls " pu3 =A0;
>
> > end-module
>
> > \ Ex. 5: Access private member of Bar
>
> > ' Bar>private>order
> > cr .( Private member of Bar: ) pr4 . cr
>
> > previous
>
> > -------------------------------------------
> > \ Now, the interface
>
> > ( =A0 =A0 =A0Title: =A0Modular programming in Forth
> > =A0 =A0 =A0 =A0 =A0File: =A0module.fs
> > =A0 =A0 =A0 Revised: =A0October 5, 2011
>
> > Description:
> > -----------
>
> > This library provides a facility for modular programming in
> > Forth. It supports both named and anonymous modules, and allows
> > for private definitions in a module to be made accessible.
> > The library uses design features from several sources:
> > David N. Williams' root-module.fs, Bruce McFarling's suggested
> > module facility for the Forth Scientific Library [FSL], the
> > original module facility in the FSL's fsl-util auxiliary files, and
> > Neal Bridges' anonymous modules code.
>
> > The layout of a module is determined by the following words:
>
> > =A0 =A0MODULE: =A0BEGIN-MODULE =A0PUBLIC: =A0PRIVATE: =A0END-MODULE
>
> > Here, BEGIN-MODULE is where the module begins, and END-MODULE
> > is where it ends. =A0The code area between the two is "inside" the
> > module. The words PUBLIC: and PRIVATE: control the visibility
> > of the module code to external code and to the user. The use of
> > MODULE: , to provide a named reference to the module, is
> > optional.
>
> > Terminology:
> > -----------
>
> > 1. The term "anonymous module" is a module which is
> > =A0 =A0 unnamed.
>
> > Design elements:
> > ---------------
>
> > 1. A module may or may not be named:
>
> > =A0 =A0 a] For an unnamed [anonymous] module, the public definitions
> > =A0 =A0 =A0 =A0are compiled into the current compilation wordlist.
>
> > =A0 =A0 b] For a named module, the public definitions are compiled
> > =A0 =A0 =A0 =A0into a new wordlist. The module name may be used
> > =A0 =A0 =A0 =A0to add the public wordlist to the search order, for use
> > =A0 =A0 =A0 =A0by other modules or external code.
>
> > =A0 =A0 c] For both a named and anonymous module, a new private
> > =A0 =A0 =A0 =A0wordlist is created. For named modules, the private
> > =A0 =A0 =A0 =A0wordlist id can be retrieved using>PRIVATE on the
> > =A0 =A0 =A0 =A0module xt. The private wordlist of an anonymous module
> > =A0 =A0 =A0 =A0is not accessible outside the module.
>
> > 2. The module layout is not nestable.
>
> > 3. The search order depth and compilation wordlist just before
> > =A0 =A0 BEGIN-MODULE are restored by END-MODULE.
>
> > 4. PRIVATE: and PUBLIC: should occur only inside a module, and may
> > =A0 =A0 be invoked any number of times there, in any order. =A0Definiti=
ons
> > =A0 =A0 inside a module usually go into its public or private
> > =A0 =A0 wordlist.
>
> > 5. The part of the search order present just before the
> > =A0 =A0 beginning of a module may not be changed inside the module.
>
> > =A0 =A0 [This could be made safer by actually saving and restoring
> > =A0 =A0 the initial search order. =A0This implementation does not do
> > =A0 =A0 that.]
> > )
>
> > \ *** GENERAL USE
>
> > [UNDEFINED] drops [IF] =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 \ BMcF
> > : drops ( +n -- ) 0 ?DO drop LOOP ; [THEN]
>
> > [UNDEFINED] order-drops [IF]
> > : order-drops =A0( +n -- ) ( o: wid_n ... wid_1 -- )
> > =A0 =A00 ?DO previous LOOP ; [THEN]
>
> > [UNDEFINED] order-depth [IF]
> > : order-depth =A0( -- n ) =A0get-order dup>r drops r> =A0; [THEN]
>
> > [UNDEFINED]>order [IF] =A0 =A0 =A0\ same as BMcF's ADD-WORDLIST
> > :>order =A0( wid -- order: wid )
> > =A0 =A0>r get-order r> =A0swap 1+ set-order ; [THEN]
>
> > [UNDEFINED] order> =A0[IF] =A0\ not used in this module
> > : order> =A0 ( o: wid -- ) ( -- wid )
> > =A0 =A0get-order swap>r 1- set-order r> =A0; [THEN]
>
> > [UNDEFINED] order@ [IF] =A0\ not used in this module
> > : order@ =A0( o: wid -- ) ( -- wid )
> > =A0 =A0get-order over>r drops r> =A0; [THEN]
>
> > VARIABLE initial-defs =A0 =A0 =A0 =A0 get-current initial-defs !
> > VARIABLE initial-order-depth
> > VARIABLE private-defs =A0 =A0 =A0 =A0 get-current private-defs !
> > VARIABLE public-defs =A0 =A0 =A0 =A0 =A0initial-defs @ public-defs !
> > VARIABLE named-module =A0 =A0 =A0 =A0 false named-module !
>
> > \ Named module
>
> > [UNDEFINED] MODULE: [IF]
> > \ define an extended vocabulary word for a named module
> > : MODULE: =A0( "name" -- )
> > =A0 =A0 =A0get-current =A0initial-defs !
> > =A0 =A0 =A0true =A0 =A0 =A0 =A0 named-module !
> > =A0 =A0 =A0\ create the private and public wordlists
> > =A0 =A0 =A0wordlist wordlist 2dup CREATE , ,
> > =A0 =A0 =A0set-current =A0private-defs !
> > =A0 =A0DOES> =A0 ( o: -- wid ) =A0@>order =A0\ add public wordlist to s=
earch
> > order
> > ;
> > [THEN]
>
> > \ Return private wordlist id from the xt of a named module
> > :>PRIVATE ( xtmodule -- wid )>body cell+ @ ;
>
> > \ *** PUBLIC DEFINITIONS
>
> > \ public-defs @ set-current =A0\ same as initial-defs
>
> > \ Defns of PRIVATE: and PUBLIC: are equivalent to BMcF's
> > : PRIVATE: =A0( -- ) =A0private-defs @ set-current ;
> > : PUBLIC: =A0 ( -- ) =A0public-defs =A0@ set-current ;
>
> > \ BEGIN-MODULE and END-MODULE are not to be nested.
>
> > : BEGIN-MODULE =A0( -- ) ( o: -- public private )
> > (
> > Save information that allows END-MODULE to restore the initial
> > search order and compilation wordlist, assuming the initial
> > order has remained unchanged. =A0Define "name" as a vocabulary
> > word belonging to the MODULES wordlist, and add its wordlist,
> > "public", to the search order. =A0Create an anonymous wordlist,
> > "private", add it to the search order, and make it the
> > compilation wordlist. =A0Store the public and private wids for use
> > by PUBLIC: and PRIVATE:.
> > )
> > =A0 =A0order-depth =A0initial-order-depth !
> > =A0 =A0get-current =A0dup>order public-defs !
> > =A0 =A0named-module @ if
> > =A0 =A0 =A0private-defs @
> > =A0 =A0else
> > =A0 =A0 =A0public-defs @ initial-defs !
> > =A0 =A0 =A0wordlist dup private-defs !
> > =A0 =A0then
> > =A0 =A0dup>order
> > =A0 =A0set-current =A0 =A0\ default section is private
> > ;
>
> > : END-MODULE =A0( o:<extras> =A0-- )
> > (
> > Restore the initial search order depth and compilation wordlist. =A0An
> > ambiguous condition exists if the initial search order has been
> > clobbered.
> > )
> > =A0 =A0initial-defs @ set-current
> > =A0 =A0order-depth initial-order-depth @ - order-drops
> > =A0 =A0false =A0named-module ! ;
>
> > \ --------------------------------------------
>
> > \ The examples below assume that module.fs is loaded.
>
> > \ EXAMPLE 1: Layout for anonymous module
> > 0 [IF]
>
> > BEGIN-MODULE
> > \ Add named modules and wordlists to the search order on which
> > \ this module depends. =A0They will be removed from the top of the
> > \ search order By END-MODULE. =A0Keeping the list of modules on
> > \ which it depends inside a module more clearly documents its
> > \ dependencies, whereas adding modules before BEGIN-MODULE:
> > \ makes it easier to introduce inadvertent, unwanted
> > \ dependencies.
>
> > <mod_1> =A0...<mod_n>
>
> > \ Initial definitions are private, and placed in a hidden
> > \ =A0 wordlist accessible only to this module.
>
> > <initial private definitions>
>
> > PUBLIC:
> > \ public definitions go into the current compilation wordlist
>
> > \ any number of subsequent PRIVATE: and PUBLIC: sections follow,
> > \ in any order.
>
> > END-MODULE
> > [THEN] =A0\ END EXAMPLE 1 LAYOUT
> > \ --------------------------------------------
>
> > \ EXAMPLE 2: Layout for a named module
> > 0 [IF]
>
> > \ Make a vocabulary for the public definitions of a
> > \ module named Foo.
>
> > Module: Foo
>
> > BEGIN-MODULE
>
> > \ Add named modules and wordlists to the search order on which
> > \ this module depends.
> > <mod_1> =A0...<mod_n>
>
> > PUBLIC:
> > \ public definitions go into the wordlist, Foo.
>
> > \ any number of subsequent private and public sections follow,
> > \ in any order.
>
> > END-MODULE
> > [THEN] =A0\ END EXAMPLE 2 LAYOUT
> > \ --------------------------------------------
>
> > \ To access the private definitions of Foo from another
> > \ module, retrieve the private wordlist of Foo, and add it
> > \ to the search order, e.g.
> > \
> > \ =A0 ' Foo>private>order
>
> > --------------------------------
>
> I've been following the discussion with some interest and, while I have
> sort of lost track of where different people stand on the topic, it
> seems to me that a consensus is not emerging. ...

Yes, I also do not see a clear consensus emerging from this
discussion.

> ... I am increasingly feeling
> that it might be better to split responsibilities for modularisation of
> code between the modules and the user.
>
> For example if there were 4 words available for use by a module:
>
> BEGIN-MODULE which saves the compilation wordlist and top wordlist in
> the search order as the public and private wordlists respectively,
> setting the private wordlist as the compilation wordlist.
>
> PRIVATE and PUBLIC setting their wordlists as the compilation wordlist.
>
> END-MODULE restores the original compilation wordlist
>
> Then it would be up to the user to decide whether to name a module or
> not, whether to create a new public wordlist or not, whether to have a
> private wordlist (use the public one for both if a flat namespace is
> wanted) and so on.
>
> The user sets the strategy, the module does the mechanics.
>

That's certainly a partial solution, but really only useful for
anonymous modules. It means that a module cannot declare its
dependencies on other modules -- one of the benefits provided by named
modules. The user of the modules has to set up dependencies for the
search order prior to loading each module, and then removing the
dependencies from the search order. This could become extremely
tedious. In contrast, DNW's root-module.fs, and my module.fs, provide
a very convenient way of automatically bringing dependencies into the
search order through the module names, and automatically removing them
from the search order.

Krishna

Gerry Jackson <gerry@jackson9000.fsnet.co.uk> wrote:
> 
> I've been following the discussion with some interest and, while I have 
> sort of lost track of where different people stand on the topic, it 
> seems to me that a consensus is not emerging. I am increasingly feeling 
> that it might be better to split responsibilities for modularisation of 
> code between the modules and the user.
> 
> For example if there were 4 words available for use by a module:
> 
> BEGIN-MODULE which saves the compilation wordlist and top wordlist in 
> the search order as the public and private wordlists respectively, 
> setting the private wordlist as the compilation wordlist.
> 
> PRIVATE and PUBLIC setting their wordlists as the compilation wordlist.
> 
> END-MODULE restores the original compilation wordlist
> 
> Then it would be up to the user to decide whether to name a module or 
> not, whether to create a new public wordlist or not, whether to have a 
> private wordlist (use the public one for both if a flat namespace is 
> wanted) and so on.
> 
> The user sets the strategy, the module does the mechanics.

I'm not sure that will exactly work because modules need to declare
their dependencies on other modules.  However, it's a great idea in
principle.

Andrew.

On Oct 10, 10:30=A0am, Krishna Myneni <krishna.myn...@ccreweb.org>
wrote:
> On Oct 9, 7:29=A0pm, jacko <jackokr...@gmail.com> wrote:

> > ...
> > What should modules achieve.
> > ...

> A very good question, and one to which there will probably be no
> uniform reply. However, I can give you my take on what programming
> facility should be provided by modules. Also, given that there are
> several existing implementations (FSL, Neal Bridges' modules.x, David
> Williams' root-modules.fs, Swift Forth's package facility, etc.),
> there is a need for some form of enhanced ordering/control of
> compilation wordlist and search order in Forth.

> Here are my expectations for a modules facility in Forth.

> 1) It should allow a group of definitions related to a single task to
> be organized into a distinctly named entity, called the module.

For what purpose? Expectations in terms of what problems it solves and
what problems it avoids would be useful here.

Organizing the definitions into a distinct include file accomplishes
everything up to "called a module".

Calling this an "activity", (1) would be served by:

: ACTIVITY: ( "name" -- ) \ "name" ( o: -- activity-wid )
   CREATE WORDLIST ,
   DOES> @ >R GET-ORDER R> SWAP 1+ SET-ORDER ;

.... but I extend the semantics below so that "ACTIVITY: name"
uniformly leaves the activity in the search order and as the
compilation target, whether or not

> 2) The definitions within a module should fall into two groups: public
> and private. The public definitions constitute the interface to code
> external to the module, while the private definitions support the
> public definitions, but are not needed, and therefore not visible, to
> external code.

Note that whether the private definitions are ever needed by external
code is better judged by the potential user than by the module author.

> Private definitions are isolated from being referenced,
> and therefore, from being modified, by external code. An exception
> mechanism allows private definitions to be made visible for debugging
> from the interpreter, and also for rare cases when direct access to
> private definitions by external code is more convenient than a public
> interface. In general, however, public definitions should provide all
> of the necessary interface to interact with the module.

Note that the description assumes that each set of public words
associated with an activity ought to have exactly one private
namespace tied to it.

The only thing that makes the need to go into a private namespace a
relatively "rare case" in Forth is the paucity of useful publicly
accessible libraries. If there were more, the need to have a way to
tinker with the details of a module for integration into a setting
that the authors had not anticipated, including for integration with
*other* useful publicly accessible libraries, would be far more
pressing.

> 3) The module can be addressable by name in order to explicitly bring
> its definitions into the search order, for use by subsequent modules
> or other external code which make use of its public interfaces.

The key phrase here is "can be", which makes it sound like its
*adding* a facility that is not already present, as opposed to
*mandating* one of several possible competing approaches.

No special provision needs to be made to make (3) *possible*. Special
provision is only required to make it *mandatory*.

> 4) The modular programming facility should make it easy to load
> modules into the system without adverse side effects to the
> compilation wordlist and search order. That is, it provides some
> guarantees on the end state of the system, with respect to the
> compilation wordlist and search order, after a module has been loaded,
> thereby relieving the programmer of having to worry about resetting
> the state of the system after each load.

Now *this* at least implicitly address an acutal *problem* to be
resolved, which is loading an include file that messes with the search
order at the time called.

Problems to be solved:

(1) load modules into the system without adverse side effects to the
compilation wordlist and search order.

This is a high priority problem to solve on the library side, whether
it is a priority on the application side depends on approach ~ "ignore
initial compilation wordlist and start each application module from a
known starting point" also works on the application side, where it
doesn't on the library side.

(2) Avoid name conflicts when compiled into an arbitrary namespace.
This is a priority to solve on the library side, and sometimes a
priority while at other times no issue on the application side as name
conflicts are eliminated at an integration step, when the target
namespace is in no way arbitrary.

(3) Provide a system for accessing those named wordlists that are
provided. On the library side, this is solving a problem introduced
when resolving (2), and if it is unnecessary to access named wordlists
at all for basic uses of the library, that is clearly superior. On the
application side, how much recourse to named wordlists, and the
implementation strategy taken to providing named wordlists, is partly
problem driven and partly driven by the preferences of the author or
authorship team.

Given the above, the following is a more flexible system than the
solution *assumed into* the above specification of what a modular
programming system ought to look like.

0 [IF]
\ the private system below uses self-pushing wordlists
\ called "activities".

ACTIVITY: foo \ foo is now in search order and is current
              \ foo is created if it does not exist
.... (arbitrary search order manipulations may follow.

foo \ foo is now in search order
[THEN]

: namespace: ( "name" -- )
   CREATE WORDLIST ,
   DOES> @ >R GET-ORDER R> SWAP 1+ SET-ORDER ;

: ACTIVITY: ( "name" -- ) ( o: -- name-wid )
   >IN @ POSTPONE [UNDEFINED] IF
      DUP >IN ! namespace:
   THEN >IN ! ' EXECUTE DEFINITIONS ;

0 [IF]
 BEGIN-MODULE ( search order state saved )
 ... ( arbitrary search order manipulations, above original order )
 PUBLIC ( original current restored )
 ... ( arbitrary search order manipulations )
 PUBLIC ( original current restored
 ...
 END-MODULE
[THEN]

GET-CONTEXT

ACTIVITY: MODULES

[UNDEFINED] drops [IF]
 : drops ( xn .. x1 n -- ) 0 ?DO DROP LOOP ;
[THEN]
[UNDEFINED] >order [IF]
 : >order ( wid -- ) ( o: -- wid )
    >R GET-ORDER R> SWAP 1+ SET-ORDER ;
[THEN]
[UNDEFINED] order> [IF]
 : order> ( -- wid ) ( o: wid -- )
   GET-ORDER SWAP R> 1- SET-ORDER R> ;
[THEN]
[UNDEFINED] order@ [IF]
 : order@ ( -- wid ) ( o: wid -- )
   GET-ORDER OVER R> drops R> ;
[THEN]
[UNDEFINED] order-depth [IF]
 : order-depth ( -- n ) GET-ORDER DUP >R drops R> ;
[THEN]
[UNDEFINED] order-drops [IF]
 : order-drops ( n -- ) ( o: wid1 .. widn -- )
    0 ?DO PREVIOUS LOOP ;
[THEN]

( compilation-wid ) VARIABLE public-wid public-wid !
VARIABLE prior-depth order-depth 1- prior-depth !

: PUBLIC ( -- ) public-wid @ SET-CONTEXT
: END-MODULE ( -- ) PUBLIC order-depth prior-depth @ - order-drops ;

PUBLIC

: BEGIN-MODULE ( -- )
   GET-CONTEXT public-wid ! order-depth prior-depth !
   MODULES ;

END-MODULE

0 [IF]
When swapping back and forth between defining words into
a private activity and defining public interface words, it
is convenient to do the switch to the private activity
with a single compilation target switch, rather than:
   activity-name order> SET-CONTEXT

PRIVATE-ACTIVITY: creates the activity if it does not
exist in the current search order, and then in any
event sets it up as the PRIVATE compilation target.

Just as with ACTIVITY:, the activity created by
PRIVATE-ACTIVITY: is in the search order and is the
compilation target when it completed.
[THEN]

BEGIN-MODULE DEFINITIONS

VARIABLE private-wid MODULES order> private-wid !

: PRIVATE ( -- ) private-wid @ SET-CONTEXT ;

PUBLIC

\ private-activity ensures name-wid is on top of search order
\ it does not add it if it is already there
: PRIVATE-ACTIVITY: ( "name" ) ( o: -- name-wid )
   ACTIVITY: order@ private-wid !

 ;

END-MODULE

0 [IF] These simplified modules are *much* more flexible, so they are
much more useful to nest. For instance, one might want to can have a
loadscript with:
  BEGIN-MODULE
   ( ... set up module hierarchy )
  module-include moduleA.fs
   ( ... more search order manipulations )
  module-include moduleB.fs
   ( ... more search order manipulations )
  module-include moduleC.fs
  END-MODULE
.... and allow for BEGIN-MODULE and END-MODULE in the filescripts being
included. Here private and public is not an issue, since that is for
filescript, not the loadscript, but its still convenient to restore
the search order to the way it was when the loadscript launched.
[THEN]

BEGIN-MODULE DEFINITIONS

: module> ( -- x1 ... xn n )
   private-wid @ public-wid @ prior-depth @ 3 ;

: >module ( x1 ... xn n )
   3 <> ABORT" Unrecognized module tag."
   prior-depth ! public-wid ! private-wid ! ;

: module-include ( "filename" -- )
   module> INCLUDE >module ;

END-MODULE

Points 3 and 4 are addressed by using a vocabulary chain, instead of a tree=
 (occasionally useful), and taking advantage of the non immediate nature of=
 vocabularies to make a word to chain forwards to a useful vocabulary. The =
backwards chain being just FIND chain following.

If you find yourself needing to switch backwards and forwards between vocab=
ularies within a definition, then perhaps a local alias should be created. =
Blank vocabularies can be a powerful word navigation tool.

Maybe this can be assisted by having vocabularies remember the last context=
, and having a word to BUMP down a context (effect a restore to last contex=
t), but this is still an independent idea from having a search order. (a.k.=
a. scoping). As this is not a true stack, for example FORTH FORTH will alwa=
ys BUMP to FORTH it maybe has some faults.

The question of the need for a search order for ease of compilation and nav=
igation versus the strictness of scope and spelling error close matches and=
 resultant difficult to find bugs. Perhaps if module definitions collide in=
 the namespace, then a module is in order to define the preferred arbitrati=
on. Littering source with many pre-qualifiers (module specifiers) and exit =
context specifiers (must be included without some create does> thing), make=
s for untidy code. Especially if the module name has a version number attac=
hed.

Cheers Jacko

On Oct 10, 11:24=A0am, jacko <jackokr...@gmail.com> wrote:
> Using a vocabulary word also had to remove itself from the search order b=
efore placing it at the front in order to prevent a circular search order, =
due to only having one link pointer. This of course had effects on restorin=
g CURRENT or CONTEXT, as the vocabulary could have been excised from the se=
arch order.

Quite. As with many minimal solutions, some care needs to be taken in
its use.

That's why, though the set-up permits PREVIOUS, I'd suggest that the
application side use of that type of vocabulary system be restricted
to lines along the line of:
  /FORTH vocab1 vocab2 DEFINITIONS vocab3

.... that is, if it also starts with the F79 "FORTH" reset, the risk of
circular search orders can be eliminated.

> This minor bug is livable in most situations, and probably preferred to a=
 full search order mash up.

> In terms of modules, this bug a.k.a feature would remove the private defi=
nitions vocabulary from the search order, and so be a kind of Zen, if it we=
re not for the fact the vocabulary would by means of creation be fore most =
in the search order.

I don't know what this intends to say.

> But it would remove any nested module privates from the search order too.=
 It could perhaps remove FORTH or any other referenced within a module voca=
bulary, which would be bad, bad, bad!

In this setting, inheriting the public wordlist and having a named
private wordlist created by the module would be the safest way to add
code that does *not* build its CONTEXT starting from /FORTH ...
indeed, perhaps the only safe way ... the named private wordlist could
not *be* lower down in the search order, precisely because it is
created by the module.

Here, I assume that ACTIVITY: creates the vocabulary if required, and
in any event make the vocabulary the CURRENT and CONTEXT.

/FORTH CONTEXT @ CURRENT @

ACTIVITY: MODULES

VARIABLE prior-context
VARIABLE public-words
VARIABLE private-words

public-words ! prior-context !

: PRIVATE ( -- ) private-words @ CURRENT ! ;
: PUBLIC ( -- ) public-words @ CURRENT ! ;
: END-MODULE ( -- ) prior-context CONTEXT ! PUBLIC ;

PUBLIC

: BEGIN-MODULE: ( "name" )
   CONTEXT @ prior-context !
   CURRENT @ public-words !
   ACTIVITY:
   CURRENT @ private-words ! ;

END-MODULE

> Hence the CONTEXT hack. Better an aphasic forth than a vegetable brick! H=
ence the always check FORTH last idea. With the modules/units idea the mult=
iple vocabs on the search order disappears though. So why need a search ord=
er?

For self-executing data files, for controlling host and target
compilation targets in a cross compiler, for namespace control when
using an object oriented programming model, etc.

> One CONTEXT is enough. Premature optimization is the root of all complexi=
ty in implementation of structured programming words ;P

The impetus behind the Forth94 search order is not optimizing task
specific vocabularies ~ that is rather the Forth83 search order. The
impetus behind the Forth94 search order is that task-specific
vocabularies is just one specialized application, and a more general
solution that is adequate to all the other namespace control needs
will intrinsically be sufficient if someone should desire a task-
specific vocabulary.

However, a minimal free standing Forth might not require those other
capacities. If it also wished to have task specific namespaces, the
extended version of the Forth79 semantics on the application side
combined with PUBLIC / PRIVATE modules to load modular code would
likely suffice.

To do one-off searches in closed wordlists, one would require a
pattern like:
 ... CONTEXT @
   ListA CONTEXT ! BL WORD FIND IF ... ELSE ... THEN
 CONTEXT ! ...

On Oct 10, 12:49=A0pm, jacko <jackokr...@gmail.com> wrote:
> If you find yourself needing to switch backwards and forwards between voc=
abularies within a definition, then perhaps a local alias should be created=
..

When compiling, if vocabularies are used for visibility control, its
more common to need to switch CURRENT back and forth than to have to
switch the CONTEXT back and forth. In a modular context, CONTEXT just
needs to be reset at the end of the module to what it was at the
beginning.

On Oct 10, 11:04=A0am, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
wrote:

> I've been following the discussion with some interest and, while I have
> sort of lost track of where different people stand on the topic, it
> seems to me that a consensus is not emerging.

Positions versus implementation details are clearer.

Whether to inherit the PUBLIC compilation wordlist or to have the
module required to create one seems to be the main position. The
former is Forth Inc's solution for Swiftforth, and I believe is Andrew
Haley's position and mine, the latter is Krishna's position, and maybe
David N Williams (?).

> I am increasingly feeling
> that it might be better to split responsibilities for modularisation of
> code between the modules and the user.
>
> For example if there were 4 words available for use by a module:

> BEGIN-MODULE which saves the compilation wordlist and top wordlist in
> the search order as the public and private wordlists respectively,
> setting the private wordlist as the compilation wordlist.

This is where application side and library side comes into play.
Library side, one wants to restore the search order to what it was
before this file started messing with it. That points against having
the private wordlist set up *outside* of BEGIN-MODULE.

At the same time, that save search order state, fool around, and reset
it is useful independent of whether there are PRIVATE and PUBLIC
sections.

Now, if you don't *have* PRIVATE sections, you still may want to
export definitions back to the original compilation wordlist, hence:
   BEGIN-MODULE PUBLIC and END-MODULE
.... as a core.

The most flexible approach to private wordlists is to *nominate* a
vocabulary as a private wordlist, creating it if it doesn't exist.
Hence:
   PRIVATE-ACTIVITY: and PRIVATE

> PRIVATE and PUBLIC setting their wordlists as the compilation wordlist.

> END-MODULE restores the original compilation wordlist

Though, as above, should restore the search order as well.

> Then it would be up to the user to decide whether to name a module or
> not, whether to create a new public wordlist or not, whether to have a
> private wordlist (use the public one for both if a flat namespace is
> wanted) and so on.

> The user sets the strategy, the module does the mechanics.

And for an external source library, the library may do PRIVATE stuff
according to its own system, but it leaves the user to set the
strategy for the PUBLIC stuff.

On Oct 10, 12:31=A0pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> I'm not sure that will exactly work because modules need to declare
> their dependencies on other modules. =A0However, it's a great idea in
> principle.

Does that have to be done with a module-specific system? If it is
incumbent on users to load various modules from the same library into
the same public target, in return for their flexibility in setting up
that target where and how they wish, then:

\ private words from foo1 are used, only public foo2 is needed.
REQUIRE foo1.fs
REQUIRE foo2.fs

BEGIN-MODULE foo1 PRIVATE-ACTIVITY: bar
   ...
END-MODULE

.... would suffice.

: PAIR CONTEXT @ CURRENT @ VOCABULARY DEFINITIONS VOCABULARY CURRENT ! CONTEXT ! ;

The above word is useful for making a pair of mutually accessible vocabularies. One does the access by being defined after and having the first in the search after chain, while the other contains the second.

This or something similar is the best way of performing a split of definitions, as each would appear to the other as equals, in terms of switching between them, even though an exact symmetry of construction is by nature impossible.

PAIR LEFT RIGHT

Construction of an N way split is more complex.

Cheers Jacko

On Oct 10, 5:12=A0pm, jacko <jackokr...@gmail.com> wrote:
> This or something similar is the best way of performing a split of defini=
tions, as each would appear to the other as equals, ...

I've never seen a need to create two vocabularies, with each appearing
to the other as an equal.

On Oct 10, 11:24=A0am, jacko <jackokr...@gmail.com> wrote:
> Using a vocabulary word also had to remove itself from the search order b=
efore placing it at the front in order to prevent a circular search order, =
due to only having one link pointer.

> This of course had effects on restoring CURRENT or CONTEXT, as the vocabu=
lary could have been excised from the search order.

Also, AFAIU, if the link through between the vocabularies is done as a
*stack* of nul dictionary entries ~ then you can scan that stack from
top down for an already existing copy, either find one and snip it out
or get to the bottom of the stack, and then support PREVIOUS that
restores the prior version if it existed.

That really is a search order stack, even if implemented as a stack of
nul dictionary entries to string vocabularies together,so while it
would be a slower "GET-ORDER" and "SET-ORDER" than a simple stack of
pointers to the address of the end of wordlist address, a Forth94
search order wordset could be added to that system after the fact.

It's a prerequisite for some types of scoping, when the exports appear named by the module.

Makes me wonder if a hard dictionary boundary is useful. By this I mean a d=
ictionary contains only its own name to start, and does not by design have =
a link back to the vocabulary it was defined in. This is the reason I think=
, FORTH is always searched last, so that ONLY and ALSO are always available=
, and vocabs not in scope can be found by both of them being prefix words. =
The construct ONLY FORTH ALSO ANOTHER is slightly silly though, as FORTH is=
 always searched, so ONLY ANOTHER would, or should suffice.

Cheers Jacko

On Oct 10, 3:41=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > On Oct 9, 12:38?pm, BruceMcF <agil...@netscape.net> wrote:
> >> On Oct 9, 9:09?am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> >> > Just so I didn't misunderstand your statement above. I'm under
> >> > the impression that everyone (in this discussion) agrees that
> >> > executing the name of a module pushes its *public* wordlist into
> >> > the search order.
>
> >> No, after digesting some of the objections to modular programming at
> >> all, I prefer a library module system to follow the SwiftForth
> >> approach where the PUBLIC wordlist *is* the compilation wordlist when
> >> the PACKAGE is called, and the PRIVATE wordlist is what is named by
> >> the PACKAGE.
> >> ...
>
> > Ah, so the only point of agreement I thought we had, really isn't.
> > Offhand I don't agree that the module name should provide the private
> > wordlist.
>
> Surely this should be up to the includer of a library. =A0If you really
> want a library's public definitions to be in a separate wordlist you
> can do, e.g.
>
> vocabulary trigonometry
> also trigonometry definitions
> include trigonometry.f
>
> Andrew.

Sure you can do this with VOCABULARY, or even with WORDLIST (since
VOCABULARY is not standard). But you haven't ensured the dependencies
for trigonometry.f are present in the search order, and that
subsequent code gets compiled into the proper wordlist. There's no way
to recover the previous compilation wordlist with what you have
written.

The modules code can be thought of simply as a convenient
factorization of the mechanics needed to do wordlist creation, search
order handling, and compilation wordlist handling that is typical in
these situations.

Krishna

On Oct 10, 7:06=A0pm, jacko <jackokr...@gmail.com> wrote:
> It's a prerequisite for some types of scoping, when the exports appear na=
med by the module.

But being able to export to a named vocabulary does not require that
the vocabulary be created in the module, and it can be convenient if
it explicitly need not be.

As previously, an EXPORT switch for CURRENT is dead simple if the user
is responsible for avoiding closed loops ~ a little more involved if
the EXPORT-ACTIVITY: word had to ensure that closed loops are avoided.
That semantic will of course work if a search order stack is
available.

On 10/10/11 12:48 PM, BruceMcF wrote:
 > On Oct 10, 10:30 am, Krishna Myneni<krishna.myn...@ccreweb.org>
 > wrote:
 >> [...]
 >> 2) The definitions within a module should fall into two groups: public
 >> and private. The public definitions constitute the interface to code
 >> external to the module, while the private definitions support the
 >> public definitions, but are not needed, and therefore not visible, to
 >> external code.

That's the way I've been thinking about it.

 > Note that whether the private definitions are ever needed by external
 > code is better judged by the potential user than by the module author.

I don't see it as either/or.

 >> Private definitions are isolated from being referenced,
 >> and therefore, from being modified, by external code. An exception
 >> mechanism allows private definitions to be made visible for debugging
 >> from the interpreter, and also for rare cases when direct access to
 >> private definitions by external code is more convenient than a public
 >> interface. In general, however, public definitions should provide all
 >> of the necessary interface to interact with the module.
 >
 > Note that the description assumes that each set of public words
 > associated with an activity ought to have exactly one private
 > namespace tied to it.
 >
 > The only thing that makes the need to go into a private namespace a
 > relatively "rare case" in Forth is the paucity of useful publicly
 > accessible libraries. If there were more, the need to have a way to
 > tinker with the details of a module for integration into a setting
 > that the authors had not anticipated, including for integration with
 > *other* useful publicly accessible libraries, would be far more
 > pressing.

I'm not sure what's involved here.

1) I assume the public interface, i.e., the specification of
existing public words, to be fixed, and that we're not talking
about bug fixes or optimizations that don't change the
interface.  Otherwise you would presumably either declare the
the old interface obsolescent, or make a new module.

2) If you're making a new upwardly compatible version of a
module by adding more public words and expanding the interface,
you can tinker at will with the source for the private words, no
mechanism needed for revealing them.  If you're making distinct
upwardly compatible versions, they would have to be distinct
modules.

3) If you're writing a new module that depends on independent
modules or other resources with overlapping names, you could
resolve the overlaps in its private section.

4) If you're writing an application or new module that would
benefit from private words in modules, you can just copy in
their source, and of course tinker with it.

One of the things I'm not addressing here is "interactive
exploration".  To what extent is that a substitute for missing
source, tests, and documentation?  Surely the very concept of a
module demands rigorous documentation and tests?  Modules
written in the future aren't going to have that defect, right?
:-)

-- David

On Oct 9, 8:24=A0am, Josh Grams <j...@qualdan.com> wrote:
> BruceMcF wrote:
> > \ =A0: NAMESPACE: ( "name" -- ) VOCABULARY ; \ if you have F79 style
> > vocab
> > \ =A0: NAMESPACE: ( "name" -- ) VOCABULARY DOES> @ >order ; \ for F83
> > style
>
> I'm not up on this aspect of Forth history, but I *think* you have
> that backwards. =A0Forth-83 seems to have a search order, so I'm
> guessing it was Forth-79 which had the upward-linked tree thingy.

To be more direct and less involved in archaic details, the named
Forth79 vocabulary added itself in front of FORTH, or replaced another
named vocabulary.

So what is remarked as "F79 style vocabulary" is more properly, "F79
vocabulary when immediately following FORTH".

With a possibility for multiple named stacks in some form of search
order, that's an odd behavior ~ one would either do one or the other.
F83 did "just replace". However, it was not uncommon for multi-vocab
extensions of F79 take the "just push" route.

Leaving a sufficient mess of incompatibilities in existing practice
that Forth94 just stepped around the mess.

On Oct 10, 7:23=A0pm, jacko <jackokr...@gmail.com> wrote:
> Makes me wonder if a hard dictionary boundary is useful. By this I mean a=
 dictionary contains only its own name to start, and does not by design hav=
e a link back to the vocabulary it was defined in.

Why would a dictionary other than the base dictionary contain its own
name? What use is that?

Forth94 wordlists are empty when created, having nothing in them at
all, and most certainly do not link back to the vocabulary they are
defined in. They are only represented by a *name* in the vocabulary
they are defined in if provision is made to create that name.

> This is the reason I think, FORTH is always searched last, so that ONLY a=
nd ALSO are always available, and vocabs not in scope can be found by both =
of them being prefix words. The construct ONLY FORTH ALSO ANOTHER is slight=
ly silly though, as FORTH is always searched, so ONLY ANOTHER would, or sho=
uld suffice.

You forget that in the ONLY/ALSO system:
   ONLY FORTH ANOTHER
.... arrives at a search order with ANOTHER in it, but not FORTH. That
is a big difference between "FORTH" in F79 terms and "ONLY FORTH" in
F83 terms. An open Forth can be launched on a file and at the end of
the script, assuming "Launch" is in the "Application" wordlist:
   ONLY FORTH Application Launch

Similarly when opening a file as a self-executing data file, you
rarely want FORTH in the search order while the file is being
INCLUDED.

On Oct 10, 7:31=A0pm, "David N. Williams" <willi...@umich.edu> wrote:
> On 10/10/11 12:48 PM, BruceMcF wrote:
> =A0> On Oct 10, 10:30 am, Krishna Myneni<krishna.myn...@ccreweb.org>=A0> =
wrote:

> =A0>> [...]
> =A0>> 2) The definitions within a module should fall into two groups: pub=
lic
> =A0>> and private. The public definitions constitute the interface to cod=
e
> =A0>> external to the module, while the private definitions support the
> =A0>> public definitions, but are not needed, and therefore not visible, =
to
> =A0>> external code.

> That's the way I've been thinking about it.

My edit to that would be to sprinkle several "not necessarily" in
place of "not".

However, I will note part of (2) that I would only edit to add a ",",
but it then contradicts (3): "The public definitions constitute the
interface to code[,] external to the module"

> =A0> Note that whether the private definitions are ever needed by externa=
l
> =A0> code is better judged by the potential user than by the module autho=
r.

> I don't see it as either/or.

> =A0> The only thing that makes the need to go into a private namespace a
> =A0> relatively "rare case" in Forth is the paucity of useful publicly
> =A0> accessible libraries. If there were more, the need to have a way to
> =A0> tinker with the details of a module for integration into a setting
> =A0> that the authors had not anticipated, including for integration with
> =A0> *other* useful publicly accessible libraries, would be far more
> =A0> pressing.

> I'm not sure what's involved here.

AFAICT, what's involved here is the path / extend / modify of an
existing code base.

The premise is that whatever library source gets made publicly
available in some effort by an individual or small team gets added to,
eg, FLAG, possibly making use of code already up at FLAG, and then
later someone sees a way to reuse some of that code, and does so. And,
hopefully, contributes something of some use back to FLAG. And so on.


> 1) I assume the public interface, i.e., the specification of
> existing public words, to be fixed, and that we're not talking
> about bug fixes or optimizations that don't change the
> interface. =A0Otherwise you would presumably either declare the
> the old interface obsolescent, or make a new module.

1.1 Or declare a new major version of the module ~ whether a new major
version is a new module with a related name or a new version of the
same module depends in part on implementation strategy ~ a new version
conscious of the prior version may be able to use substantial sections
of already existing code if loaded on top of a system where the prior
version has already been loaded.

1.2 ~ In these kinds of projects, partial portability is not uncommon.
One patch is to take "works on systems A, B and C" ... and extend it
to include "works on System D". Quite often what will keep it from
"working on System D" is an assumption made within the private
namespace.

1.2a ~ If the original has been built with enough conditional
compilation, you build the "fixed" definitions up front, and then when
the same private-namespace is nominated, the correct versions already
exist and so the incorrect versions are not defined in the first
place.

1.2b ~ Or if not, load the original, define as much of the corrected
words as required to patch, and redefined the PUBLIC words that
require redefinition.

1.3 For extensions, it may well be a small module that REQUIRES a
larger existing module.


> 2) If you're making a new upwardly compatible version of a
> module by adding more public words and expanding the interface,
> you can tinker at will with the source for the private words, no
> mechanism needed for revealing them. =A0If you're making distinct
> upwardly compatible versions, they would have to be distinct
> modules.

2.1 ~ Tinkering with an extension that makes a new use of some set of
private words in the private namespace only requires that you
understand those specific words. Tinkering directly with the source,
first, means you have to grok the whole source to avoid messing up
some other part as you tinker, and second that each distinct module
has a independent copies of all the common words.

2.2 ~ More, though, what if the maintainer of the original gets around
to fixing a pesky bug. If the bug fix was in a different aspect of the
original module than you were building on top of, presto bango, load
the fixed source and the pesky bug your source had inherited from the
original module is fixed.

And if you were having to code around the bug, you could well be in
the position to simplify your extension to remove the work around.

> 3) If you're writing a new module that depends on independent
> modules or other resources with overlapping names, you could
> resolve the overlaps in its private section.

I don't know what this is saying.

> 4) If you're writing an application or new module that would
> benefit from private words in modules, you can just copy in
> their source, and of course tinker with it.

Then what in the hell was the purpose of putting together the neatly
packaged, hopefully tested and regression testable, modules, if you
are going to build on the modules by copy and paste and muck around?

> One of the things I'm not addressing here is "interactive
> exploration". =A0To what extent is that a substitute for missing
> source, tests, and documentation? =A0Surely the very concept of a
> module demands rigorous documentation and tests? =A0Modules
> written in the future aren't going to have that defect, right?

It may not demand rigorous documentation and tests, but it certainly
*invites* it, by increasing the payoff of the documentation and tests
by virtue of the opportunities to leverage the *use* of the modules
*as written*.

Which is why I am puzzled by the "just copy and paste and thrash
around, if you want to re-use the code". It kinds of points toward ad
hoc forking, just to do the kind of thing that with a quite simple but
thought through private namespace could straightforwardly load the
module as written and then re-use it from there.

On Oct 10, 7:25=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 10, 3:41=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
> > Surely this should be up to the includer of a library. =A0If you really
> > want a library's public definitions to be in a separate wordlist you
> > can do, e.g.

> > vocabulary trigonometry
> > also trigonometry definitions
> > include trigonometry.f

> Sure you can do this with VOCABULARY, or even with WORDLIST (since
> VOCABULARY is not standard). But you haven't ensured the dependencies
> for trigonometry.f are present in the search order, and that
> subsequent code gets compiled into the proper wordlist. There's no way
> to recover the previous compilation wordlist with what you have
> written.

Of course, the *includer* can often recover the previous compilation
wordlist just be making it the compilation wordlist again. After all,
somewhere on the application side, you have the source that decided
what that prior compilation wordlist was, and often just a little bit
above in that same file.

What is a finicky process with the order and order ext words on the
library side, where you wish to be able to compile the module into an
arbitrary wordlist on the application side while returning from the
module with the same search order as you had when you entered the
module ...
.... on the application side is often just a copy and paste of a search
order set-up line.

> The modules code can be thought of simply as a convenient
> factorization of the mechanics needed to do wordlist creation, search
> order handling, and compilation wordlist handling that is typical in
> these situations.

Yes, which is why the concern that the library side modules be done in
a way that is *convenient to be used* by the widest range of
application side systems for wordlist creation, search order handling
and compilation wordlist handling.

In layers:
Layer 1: BEGIN-MODULE {Internal: PUBLIC END-MODULE }
Layer 2: {Internal: PRIVATE-ACTIVITY: PRIVATE }
Layer 3: {Internal: EXPORT-ACTIVITY: EXPORT }

ACTIVITY: is a factor, exported as a part of the public interface as a
facility for those who may wish to use it.

0 [IF]

OK, how about this one, then.

PUBLIC is initial compilation list.

PRIVATE is anonymous unless and until a PRIVATE compilation target is
nominated ~ created if required.

EXPORT is PUBLIC unless and until an EXPORT compilation target is
nominated.

namespace: creates a self-adding list
Actions: "name" uses or creates a list, w/compilation

Private-actions: "name" uses or creates a list for Private
Export-actions: "name" uses or creates a list for Export

[THEN]

0 [IF] A simple modular programming system,
with library development extensions.

namespace:     Self-pushing vocabulary
Actions:          Add to order and compilation, make if needed
Begin-Module   Remember current compilation & order depth
MODULES        namespace containing names local to modules

Local to modules:
   PUBLIC      Compile to original compilation target
   End-Module  Restore compilation target and search depth
                     x
   Private-actions:  nominate a private compilation list
                     PRIVATE is anonymous if no list nominated
   PRIVATE     Set compilation to private list

   Export-actions:  nominate an export compilation list
                     EXPORT is PUBLIC if no list nominated
   EXPORT      Set compilation to private list

   ... & Private action lists created by Private-actions:

[THEN]

[UNDEFINED] namespace: [IF]
   : namespace: ( "name" -- ) CREATE WORDLIST ,
      DOES> ( o: -- "name"-wid )
         @ >R GET-ORDER R> SWAP 1+ SET-ORDER ;
[THEN]

GET-CONTEXT namespace: MODULES

MODULES DEFINITIONS

        [UNDEFINED] drops [IF]
   : drops ( xn ... x1 n -- ) 0 ?DO DROP LOOP ;

[THEN]  [UNDEFINED] order-drops [IF]
   : order-drops ( n -- ) 0 ?DO PREVIOUS LOOP ;

[THEN]  [UNDEFINED] order@ [IF]
   : order@ ( -- wid ) ( O: wid -- )
      GET-ORDER OVER >R drops R> ;

[THEN]  [UNDEFINED] order> [IF]
   : order> ( -- wid ) ( O: wid -- )
      GET-ORDER SWAP >R 1- SET-ORDER R> ;

[THEN]  [UNDEFINED] order> [IF]
   : >order ( wid -- ) ( O: -- wid )
      >R GET-ORDER R> SWAP 1+ SET-ORDER ;

[THEN]  [UNDEFINED] order-depth [IF]
   : order-depth ( -- n ) GET-ORDER DUP >R drops R> ;

[THEN]

: >definitions ( -- ) ( O: wid -- ) order> SET-CONTEXT ;

   \ Module state: public-wid, prior-depth
VARIABLE prior-depth
VARIABLE public-wid
VARIABLE export-wid
VARIABLE private-wid

   \ stash original compilation, & depth (already one deeper)
   \ MODULES is PRIVATE right now

order-depth 1- prior-depth !
( original-compilation ) DUP public-wid ! export-wid !
order@ private-wid !

0 [IF]
   Core layer of Modules is:
      Begin-Module ... End-Module
   which removes anything *added* to search order in between
   and restores compilation wid at time of Begin-Module

      Begin-Module PRIVATE ... PUBLIC ... End-module
   Creates an anonymous unnamed PRIVATE wordlist, and allows
   shifting compilation back and forth between PRIVATE and PUBLIC
   lists.

      Begin-Module Private-Acts: foo
         ... PUBLIC ... PRIVATE ... PUBLIC ...
      End-Module
   Ensures "foo" is on top of search order, and is PRIVATE list.
   "foo" is created in MODULES if need be.

Export words: Allows arbitrary access to a sequence of
   compilation lists, most simply when found in original
   compilation list, but also allows private lists
   hierarchies or other more complicated tasks

   Begin-Module Private-Actions: "name"
      \ "name" on top of search order and compilation
   PUBLIC
      \ original compilation list

   Export-Actions: foo
      \ "foo" on top of search order and compilation
      \ Export-Actions: "name" PREVIOUS if already in order
   PRIVATE
      \ private compilation
   EXPORT
      \ foo compilation
   Export-Actions: bar PREVIOUS
      \ bar is already in search order
   PRIVATE
      \ private compilation
   EXPORT
      \ bar compilation

[THEN]

: PUBLIC ( -- ) public-wid @ SET-CURRENT ;
: EXPORT ( -- ) export-wid @ SET-CURRENT ;

: PRIVATE ( -- )
   private-wid @ ?DUP 0= IF
      WORDLIST DUP >order DUP private-wid !
   THEN SET-CURRENT ;

: End-Module ( -- ) Public order-depth prior-depth @ -

PUBLIC

: Begin-Module ( -- ) ( O: -- MODULES-wid )
   0 private-wid !
   GET-CONTEXT DUP public-wid ! export-wid !
   order-depth prior-depth !
   MODULES ;

   \ Actions: & Export-Actions: stored in current compilation list
   \ Private-Actions: are stored in MODULES

: Actions: ( "name" -- )
   >IN @ PARSE-NAME MODULES order> SEARCH-WORDLIST 0= IF
      DUP >IN ! namespace: DUP >IN ! '
   THEN EXECUTE DEFINITIONS DROP ;

PRIVATE

: Export-actions ( "name" -- ) ( O: -- "name"-wid )
   Actions: order@ export-wid ! ;

: Private-actions: ( "name" -- ) ( O: -- "name"-wid )
   MODULES >definitions  Actions: order@ private-wid ! ;

End-Module

David N. Williams <williams@umich.edu> wrote:
> 
> 4) If you're writing an application or new module that would
> benefit from private words in modules, you can just copy in
> their source, and of course tinker with it.
> 
> One of the things I'm not addressing here is "interactive
> exploration".  To what extent is that a substitute for missing
> source, tests, and documentation?

Not at all.  The source isn't missing, and the tests and documentation
aren't necessaily going to tell you what you need to know.

When you read

: foo ( --)   bar baz ;

the next thing you're going to do is

locate bar

and if BAR is in some hidden vocabulary you've immediately got a
problem.

Andrew.

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> On Oct 10, 3:41?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>> > On Oct 9, 12:38?pm, BruceMcF <agil...@netscape.net> wrote:
>> >> On Oct 9, 9:09?am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>>
>> >> > Just so I didn't misunderstand your statement above. I'm under
>> >> > the impression that everyone (in this discussion) agrees that
>> >> > executing the name of a module pushes its *public* wordlist into
>> >> > the search order.
>>
>> >> No, after digesting some of the objections to modular programming at
>> >> all, I prefer a library module system to follow the SwiftForth
>> >> approach where the PUBLIC wordlist *is* the compilation wordlist when
>> >> the PACKAGE is called, and the PRIVATE wordlist is what is named by
>> >> the PACKAGE.
>> >> ...
>>
>> > Ah, so the only point of agreement I thought we had, really isn't.
>> > Offhand I don't agree that the module name should provide the private
>> > wordlist.
>>
>> Surely this should be up to the includer of a library. ?If you really
>> want a library's public definitions to be in a separate wordlist you
>> can do, e.g.
>>
>> vocabulary trigonometry
>> also trigonometry definitions
>> include trigonometry.f
> 
> Sure you can do this with VOCABULARY, or even with WORDLIST (since
> VOCABULARY is not standard). But you haven't ensured the dependencies
> for trigonometry.f are present in the search order,

It's up to the module to declare its dependencies.

> and that subsequent code gets compiled into the proper
> wordlist.

By definition, the current wordlist is the proper one.

> The modules code can be thought of simply as a convenient
> factorization of the mechanics needed to do wordlist creation,
> search order handling, and compilation wordlist handling that is
> typical in these situations.

I don't think it is typical, actually.  A classic dictum in Forth is
"don't bury your tools"; your proposal, of not only hiding the private
words but the public ones as well, is more extreme.  It's not so much
burying your tools as putting them in a steel coffin, welding the lid
shut, and encasing that in two metres of concrete.

Andrew.

0 [IF]
   Same sketch, except "MODULES" exists in one of the
   systems here, so I've changed it to MAKE-MODULE
   and have fixed one or two glitches
[THEN]

0 [IF] A simple modular programming system,
with library development extensions.

namespace:     Self-pushing vocabulary
Actions:          Add to order and compilation, make if needed
Begin-Module   Remember current compilation & order depth
MAKE-MODULE        namespace containing names local to modules

Local to modules:
   PUBLIC      Compile to original compilation target
   End-Module  Restore compilation target and search depth
                     x
   Private-actions:  nominate a private compilation list
                     PRIVATE is anonymous if no list nominated
   PRIVATE     Set compilation to private list

   Export-actions:  nominate an export compilation list
                     EXPORT is PUBLIC if no list nominated
   EXPORT      Set compilation to private list

   ... & Private action lists created by Private-actions:

[THEN]

[UNDEFINED] namespace: [IF]
   : namespace: ( "name" -- )  WORDLIST CREATE ,
      DOES> ( O: -- "name"-wid )
         @ >R GET-ORDER R> SWAP 1+ SET-ORDER ;
[THEN]

GET-CURRENT
namespace: MAKE-MODULE
MAKE-MODULE DEFINITIONS

   \ Module state: public-wid, prior-depth
VARIABLE prior-depth
VARIABLE public-wid
VARIABLE export-wid
VARIABLE private-wid

   \ stash original compilation

( original-compilation ) DUP public-wid ! export-wid !

        [UNDEFINED] drops [IF]
   : drops ( xn ... x1 n -- ) 0 ?DO DROP LOOP ;

[THEN]  [UNDEFINED] order-drops [IF]
   : order-drops ( n -- ) 0 ?DO PREVIOUS LOOP ;

[THEN]  [UNDEFINED] order@ [IF]
   : order@ ( -- wid ) ( O: wid -- )
      GET-ORDER OVER >R drops R> ;

[THEN]  [UNDEFINED] order> [IF]
   : order> ( -- wid ) ( O: wid -- )
      GET-ORDER SWAP >R 1- SET-ORDER R> ;

[THEN]  [UNDEFINED] order> [IF]
   : >order ( wid -- ) ( O: -- wid )
      >R GET-ORDER R> SWAP 1+ SET-ORDER ;

[THEN]  [UNDEFINED] order-depth [IF]
   : order-depth ( -- n ) GET-ORDER DUP >R drops R> ;

[THEN]

: >definitions ( -- ) ( O: wid -- ) order> SET-CURRENT ;

   \ stash depth (already one deeper) & set MAKE-MODULE as Private

CR .( order-depth is now ) order-depth . CR

order-depth 1- prior-depth !
MAKE-MODULE order> private-wid !

0 [IF]
   Core layer of Modules is:
      Begin-Module ... End-Module
   which removes anything *added* to search order in between
   and restores compilation wid at time of Begin-Module

      Begin-Module PRIVATE ... PUBLIC ... End-module
   Creates an anonymous unnamed PRIVATE wordlist, and allows
   shifting compilation back and forth between PRIVATE and PUBLIC
   lists.

      Begin-Module Private-Acts: foo
         ... PUBLIC ... PRIVATE ... PUBLIC ...
      End-Module
   Ensures "foo" is on top of search order, and is PRIVATE list.
   "foo" is created in MAKE-MODULE if need be.

Export words: Allows arbitrary access to a sequence of
   compilation lists, most simply when found in original
   compilation list, but also allows private lists
   hierarchies or other more complicated tasks

   Begin-Module Private-Actions: "name"
      \ "name" on top of search order and compilation
   PUBLIC
      \ original compilation list

   Export-Actions: foo
      \ "foo" on top of search order and compilation
      \ Export-Actions: "name" PREVIOUS if already in order
   PRIVATE
      \ private compilation
   EXPORT
      \ foo compilation
   Export-Actions: bar PREVIOUS
      \ bar is already in search order
   PRIVATE
      \ private compilation
   EXPORT
      \ bar compilation

[THEN]

: PUBLIC ( -- ) public-wid @ SET-CURRENT ;
: EXPORT ( -- ) export-wid @ SET-CURRENT ;

: PRIVATE ( -- )
   private-wid @ ?DUP 0= IF
      WORDLIST DUP >order DUP private-wid !
   THEN SET-CURRENT ;

: End-Module ( -- ) Public order-depth prior-depth @ - order-drops ;

PUBLIC

: Begin-Module ( -- ) ( O: -- MAKE-MODULE-wid )
   0 private-wid !
   GET-CURRENT DUP public-wid ! export-wid !
   order-depth prior-depth !
   MAKE-MODULE ;

   \ Actions: & Export-Actions: stored in current compilation list
   \ Private-Actions: are stored in MAKE-MODULE

: Actions: ( "name" -- )
   >IN @ PARSE-NAME GET-CURRENT SEARCH-WORDLIST 0= IF
      DUP >IN ! namespace: DUP >IN ! '
   THEN EXECUTE DEFINITIONS DROP ;

PRIVATE

: Export-actions ( "name" -- ) ( O: -- "name"-wid )
   Actions: order@ export-wid ! ;

: Private-actions: ( "name" -- ) ( O: -- "name"-wid )
   MAKE-MODULE >definitions  Actions: order@ private-wid ! ;

End-Module

On Oct 11, 4:40=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:

> I don't think it is typical, actually. =A0A classic dictum in Forth is
> "don't bury your tools"; your proposal, of not only hiding the private
> words but the public ones as well, is more extreme. =A0It's not so much
> burying your tools as putting them in a steel coffin, welding the lid
> shut, and encasing that in two metres of concrete.

The semantics where:

Begin-Module: foo

VARIABLE bar1
VARIABLE bar2

PUBLIC
: >bar ( x1 x2 -- ) bar2 ! bar1 ! ;
: bar. ( -- ) bar1 @ . bar2 @ . ;

End-Module

.... reopens or creates a named vocabulary for PRIVATE, and PUBLIC is
the original compilation wordlist, provides namespace control without
burying any tool. Its just that some tools are located inside a
labeled toolbox.

An idea and a question:

(1) The vocabulary and "order" problem.
~ using vocabulary is normally required for order to given the name,
if it exists.

But different semantics for vocabulary exist ~ over-write is the most
common, but its not universal.

If the source can detect which style of vocabulary is present, if any,
then the module system could use the *native* vocabulary, with
different versions of "Begin-Module: foo" compiling depending on which
type of vocabulary is available.

Then, application side, the private wordlists will be usable in the
native way.

(2) Should the private wordlists be looked for, and created if not
present, in the original compilation wordlist, or in the MAKE-MODULE
toolbox?

On 10/10/11 8:23 PM, BruceMcF wrote:
> On Oct 10, 7:31 pm, "David N. Williams"<willi...@umich.edu>  wrote:
>> On 10/10/11 12:48 PM, BruceMcF wrote:
>>   >  On Oct 10, 10:30 am, Krishna Myneni<krishna.myn...@ccreweb.org>
  >  wrote:
>
>>   >>  [...]
>>   >>  2) The definitions within a module should fall into two
groups: public
>>   >>  and private. The public definitions constitute the interface
to code
>>   >>  external to the module, while the private definitions support the
>>   >>  public definitions, but are not needed, and therefore not
visible, to
>>   >>  external code.
>
>> That's the way I've been thinking about it.
>
> My edit to that would be to sprinkle several "not necessarily" in
> place of "not".
>
> However, I will note part of (2) that I would only edit to add a ",",
> but it then contradicts (3): "The public definitions constitute the
> interface to code[,] external to the module"
>
>>   >  Note that whether the private definitions are ever needed by
external
>>   >  code is better judged by the potential user than by the module
author.
>
>> I don't see it as either/or.
>
>>   >  The only thing that makes the need to go into a private namespace a
>>   >  relatively "rare case" in Forth is the paucity of useful publicly
>>   >  accessible libraries. If there were more, the need to have a way to
>>   >  tinker with the details of a module for integration into a setting
>>   >  that the authors had not anticipated, including for integration
with
>>   >  *other* useful publicly accessible libraries, would be far more
>>   >  pressing.
>
>> I'm not sure what's involved here.
>
> AFAICT, what's involved here is the path / extend / modify of an
> existing code base.
>
> The premise is that whatever library source gets made publicly
> available in some effort by an individual or small team gets added to,
> eg, FLAG, possibly making use of code already up at FLAG, and then
> later someone sees a way to reuse some of that code, and does so. And,
> hopefully, contributes something of some use back to FLAG. And so on.

Okay, with due regard to the organization of dependencies on
public interfaces, which in my view means due regard for the
partial-ordering property.

>> 1) I assume the public interface, i.e., the specification of
>> existing public words, to be fixed, and that we're not talking
>> about bug fixes or optimizations that don't change the
>> interface.  Otherwise you would presumably either declare the
>> the old interface obsolescent, or make a new module.
>
> 1.1 Or declare a new major version of the module ~ whether a new major
> version is a new module with a related name or a new version of the
> same module depends in part on implementation strategy ~ a new version
> conscious of the prior version may be able to use substantial sections
> of already existing code if loaded on top of a system where the prior
> version has already been loaded.

Yes.

> 1.2 ~ In these kinds of projects, partial portability is not uncommon.
> One patch is to take "works on systems A, B and C" ... and extend it
> to include "works on System D". Quite often what will keep it from
> "working on System D" is an assumption made within the private
> namespace.
>
> 1.2a ~ If the original has been built with enough conditional
> compilation, you build the "fixed" definitions up front, and then when
> the same private-namespace is nominated, the correct versions already
> exist and so the incorrect versions are not defined in the first
> place.
>
> 1.2b ~ Or if not, load the original, define as much of the corrected
> words as required to patch, and redefined the PUBLIC words that
> require redefinition.

Thanks for spelling these things out.  It does advance the
discussion.  So far I don't *think* I disagree.

> 1.3 For extensions, it may well be a small module that REQUIRES a
> larger existing module.

Indeed, but with the reservation that REQUIRES alone is not an
adequate declaration of dependence when there have been
redefinitions of already loaded words.  Having a module name
that adds its public wordlist to the search order takes care of
that.

>> 2) If you're making a new upwardly compatible version of a
>> module by adding more public words and expanding the interface,
>> you can tinker at will with the source for the private words, no
>> mechanism needed for revealing them.  If you're making distinct
>> upwardly compatible versions, they would have to be distinct
>> modules.
>
> 2.1 ~ Tinkering with an extension that makes a new use of some set of
> private words in the private namespace only requires that you
> understand those specific words. Tinkering directly with the source,
> first, means you have to grok the whole source to avoid messing up
> some other part as you tinker, and second that each distinct module
> has a independent copies of all the common words.

Right.

> 2.2 ~ More, though, what if the maintainer of the original gets around
> to fixing a pesky bug. If the bug fix was in a different aspect of the
> original module than you were building on top of, presto bango, load
> the fixed source and the pesky bug your source had inherited from the
> original module is fixed.
>
> And if you were having to code around the bug, you could well be in
> the position to simplify your extension to remove the work around.

Yes.

>> 3) If you're writing a new module that depends on independent
>> modules or other resources with overlapping names, you could
>> resolve the overlaps in its private section.
>
> I don't know what this is saying.

What I intended is that by and large modules that do not depend
on each other shouldn't have to worry about public name
overlaps.  If a new module or application is written that
depends on two such independent modules, you can disambiguate
their overlaps in the new code, for example, via simple
synonyms, if you have the control over the search order that
module names provide (whether supplied from within or by a load
script on an anonymous module).

>> 4) If you're writing an application or new module that would
>> benefit from private words in modules, you can just copy in
>> their source, and of course tinker with it.
>
> Then what in the hell was the purpose of putting together the neatly
> packaged, hopefully tested and regression testable, modules, if you
> are going to build on the modules by copy and paste and muck around?

The above refers only to copying in private words, whose
interfaces are not guaranteed beyond having to work in the
specific versions of the modules that in which they occur.  Of
course they're tested, which can add to their usefulness as part
of a new module or application.  Once they're copied in, you're
invited to muck with them in whatever way is useful.

>> One of the things I'm not addressing here is "interactive
>> exploration".  To what extent is that a substitute for missing
>> source, tests, and documentation?  Surely the very concept of a
>> module demands rigorous documentation and tests?  Modules
>> written in the future aren't going to have that defect, right?
>
> It may not demand rigorous documentation and tests, but it certainly
> *invites* it, by increasing the payoff of the documentation and tests
> by virtue of the opportunities to leverage the *use* of the modules
> *as written*.
>
> Which is why I am puzzled by the "just copy and paste and thrash
> around, if you want to re-use the code". It kinds of points toward ad
> hoc forking, just to do the kind of thing that with a quite simple but
> thought through private namespace could straightforwardly load the
> module as written and then re-use it from there.

Again, I was referring to copying/pasting/reusing private words,
no forking intended.

-- David

On Oct 11, 11:45=A0am, "David N. Williams" <willi...@umich.edu> wrote:

> On 10/10/11 8:23 PM, BruceMcF wrote:

> > On Oct 10, 7:31 pm, "David N. Williams"<willi...@umich.edu> =A0wrote:

> > AFAICT, what's involved here is the path / extend / modify of an
> > existing code base.

> > The premise is that whatever library source gets made publicly
> > available in some effort by an individual or small team gets
> > added to, eg, FLAG, possibly making use of code already up at
> > FLAG, and then later someone sees a way to reuse some of that
> > code, and does so. And, hopefully, contributes something of
> > some use back to FLAG. And so on.

> Okay, with due regard to the organization of dependencies on
> public interfaces, which in my view means due regard for the
> partial-ordering property.

If thinking about a library side modular namespace system as being
used by more than one independent projects ~ I believe that a flat
registry supports the partially ordering.


> > 1.3 For extensions, it may well be a small module that REQUIRES a
> > larger existing module.

> Indeed, but with the reservation that REQUIRES alone is not an
> adequate declaration of dependence when there have been
> redefinitions of already loaded words.

But for a published library that has a sufficient number of published
versions to make it worth the bother, its possible to include distinct
version loadscripts that DO ensure that the required version is
available. The *library* will know how much redefinition is required
to provide for access to previous versions.

It is then an appropriate application side responsibility to ensure
that the library side cautions regarding cross-version interference is
respected.

>=A0Having a module name
> that adds its public wordlist to the search order takes care of
> that.

But it has is sufficiently inconvenient to the application side user
compared to compiling the public words directly into the original
compilation list that this is only a compelling argument if its the
*only* practicable way *to* take care of it.

And I don't think its the only practicable way to take care of it.

> >> 3) If you're writing a new module that depends on independent
> >> modules or other resources with overlapping names, you could
> >> resolve the overlaps in its private section.

> > I don't know what this is saying.

> What I intended is that by and large modules that do not depend
> on each other shouldn't have to worry about public name
> overlaps.

Whether necessary or not, its common courtesy that the public
interface words are named distinctively enough to reduce the
likelihood of public name overlaps.

But in any event, by allowing public words to inherit their
compilation wordlist, the application side user, who knows the actual
names of the actual public words being imported, can readily step
around the occasional overlap by including some of the modules into
their own toolkit and then with the target wordlist underneath and the
"quarantined public" wordlist on top of the search order, simply
defining a pseudonym in the main common wordlist.

>=A0If a new module or application is written that
> depends on two such independent modules, you can disambiguate
> their overlaps in the new code, for example, via simple
> synonyms, if you have the control over the search order that
> module names provide (whether supplied from within or by a load
> script on an anonymous module).

But you can also do that via simple synonyms if you have the control
over the compilation target that inheriting the original compilation
target as the PUBLIC compilation target provides. Its not a serious
enough application side challenge to justify making it unnecessarily
difficult to compiling as broad a sequence of public names into a
common pool as desired.

> >> 4) If you're writing an application or new module that would
> >> benefit from private words in modules, you can just copy in
> >> their source, and of course tinker with it.

> > Then what in the hell was the purpose of putting together the neatly
> > packaged, hopefully tested and regression testable, modules, if you
> > are going to build on the modules by copy and paste and muck around?

> The above refers only to copying in private words, whose
> interfaces are not guaranteed beyond having to work in the
> specific versions of the modules that in which they occur.

This goes against the dictum that Andrew reminds us of, "don't bury
your tools". A published library of modules shouldn't be going around
changing the private namespace in the published code without
documenting that fact. Now, users of the public words alone only need
to be aware of changes that affect the public words (as in a bug fix
that improves the compliance of the actual behavior to the glossary
behavior), and in a source distributed module.

Now, its quite appropriate to refer a prospective user of the private
words to the source for the details of the changes ... but if a change
in the naming or behavior of public words implies a new major version,
its a change in the private words that implies a new minor version.

>=A0Of
> course they're tested, which can add to their usefulness as part
> of a new module or application. =A0Once they're copied in, you're
> invited to muck with them in whatever way is useful.

Fine is someone wants to, but it is not an excuse to interfere with
application side loading the published version of the module and then
using all of its capacities as the user sees fit.


> >> One of the things I'm not addressing here is "interactive
> >> exploration". =A0To what extent is that a substitute for missing
> >> source, tests, and documentation? =A0Surely the very concept of a
> >> module demands rigorous documentation and tests? =A0Modules
> >> written in the future aren't going to have that defect, right?
>
> > It may not demand rigorous documentation and tests, but it certainly
> > *invites* it, by increasing the payoff of the documentation and tests
> > by virtue of the opportunities to leverage the *use* of the modules
> > *as written*.
>
> > Which is why I am puzzled by the "just copy and paste and thrash
> > around, if you want to re-use the code". It kinds of points toward ad
> > hoc forking, just to do the kind of thing that with a quite simple but
> > thought through private namespace could straightforwardly load the
> > module as written and then re-use it from there.
>
> Again, I was referring to copying/pasting/reusing private words,
> no forking intended.

The copying and pasting encourages forking, whether deliberate ~ since
the incentive to re-use as much of the original module as practicable
is stronger if the entire original module is included as a unit and is
there to be re-used ~ incremental divergence, since the tracking of
new versions is more likely in the load and adapt approach than in the
copy and paste approach ~ and unintentional forking, simply due to not
fully understanding the implication of a particular modification or
simplification performed in the process of getting the copied section
to work.

Krishna Myneni wrote:
> Below is my bare-bones sketch of an interface for modular programming
> in Forth, adapted from the code of Bruce McFarling, David N. Williams,
> and Neal Bridges. I think it provides the essential features necessary
> for modular programming, with the exceptions of a wordlist to collect
> the module references, and Bruce's HELPER words.
> ...

And here I was thinking Forth was already modular programming :)

Perhaps a more apt title would have been "Yet another name-hiding
scheme for Forth".

As such schemes go, I can't fault the one used by LMI.  No declaring
MODULE PRIVATE PUBLIC etc and the guesswork it imposes.
Just write and debug your app or 'module' as normal.  When completed
you'll be in a better position to know what words (if any) need to be hidden.
At this point declare the names (range of names in LMI) you want hidden
and you're done.  Sometimes simple really is best.

BruceMcF <agila61@netscape.net> wrote:
> On Oct 11, 4:40?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
> 
>> I don't think it is typical, actually. ?A classic dictum in Forth is
>> "don't bury your tools"; your proposal, of not only hiding the private
>> words but the public ones as well, is more extreme. ?It's not so much
>> burying your tools as putting them in a steel coffin, welding the lid
>> shut, and encasing that in two metres of concrete.
> 
> The semantics where:
> 
> Begin-Module: foo
> 
> VARIABLE bar1
> VARIABLE bar2
> 
> PUBLIC
> : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;
> : bar. ( -- ) bar1 @ . bar2 @ . ;
> 
> End-Module
> 
> ... reopens or creates a named vocabulary for PRIVATE, and PUBLIC is
> the original compilation wordlist, provides namespace control without
> burying any tool. Its just that some tools are located inside a
> labeled toolbox.

So, I'm browsing a program and I see

  : blather   ...  >bar  ... ;

and then I type

  locate >bar

and I see

  : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;

so I type

  locate bar2

I guess the next thing I'd see would be

  bar2 ?

I'd swear profusely at the author, then try to figure out which module
contained BAR2 .  If this happened more than once I'd figure out how
to disable the module system in order to make browsing easier.

I suppose, in an ideal world, LOCATE >BAR would open the module FOO
(because >BAR is exported from FOO).  I'm not sure how this would work
in practice.

Andrew.

On Oct 12, 6:41=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> BruceMcF <agil...@netscape.net> wrote:
> > On Oct 11, 4:40?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> > wrote:

> >> I don't think it is typical, actually. ?A classic dictum in
> >> Forth is "don't bury your tools"; your proposal, of not only
> >> hiding the private words but the public ones as well, is more
> >> extreme. ?It's not so much burying your tools as putting them
> >> in a steel coffin, welding the lid shut, and encasing that
> >> in two metres of concrete.

> > The semantics where:

> > Begin-Module: foo

> > VARIABLE bar1
> > VARIABLE bar2

> > PUBLIC
> > : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;
> > : bar. ( -- ) bar1 @ . bar2 @ . ;

> > End-Module

> > ... reopens or creates a named vocabulary for PRIVATE,
> > and PUBLIC is the original compilation wordlist, provides
> > namespace control without burying any tool. Its just that
> > some tools are located inside a labeled toolbox.

> So, I'm browsing a program and I see
>
> =A0 : blather =A0 ... =A0>bar =A0... ;
>
> and then I type
>
> =A0 locate >bar
>
> and I see
>
> =A0 : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;
>
> so I type
>
> =A0 locate bar2
>
> I guess the next thing I'd see would be
>
> =A0 bar2 ?

Why is locate bar2 doing that? Because its not in the namespace?

In the context of published libraries of source, if private words are
not hidden, when there is a namespace conflict among supplementary
words, wouldn't "locate bar2" give the most recently defined bar2,
which may well not be the one being used above?

> I'd swear profusely at the author, then try to figure out which module
> contained BAR2 . =A0If this happened more than once I'd figure out how
> to disable the module system in order to make browsing easier.

> I suppose, in an ideal world, LOCATE >BAR would open the module FOO
> (because >BAR is exported from FOO). =A0I'm not sure how this would work
> in practice.

If you have included bar2 from a published library, you get to the
point where you are better off opening the published library file and
seeing what it says about what bar2 is doing.

If bar2 was compiled into the open, and the private words are what is
located in "the module", locate would find bar2. If it also tells you
which file it was compiled from, you can open that file if you want
the nitty gritty on what is behind the public word.

Indeed, if there is a glossary of the public words included in the
module, and you were unfamiliar with bar2, you probably have a set of
words described there that you ought to know.

Now, you can backtrack one, look at the file that included >bar, and
then work out from the documentation or guess which of the include
files bar2 came from ... but IMHO, having the public word in the open
is better in that circumstance, since you can go directly to that file
that defined that word and its partners.

BruceMcF <agila61@netscape.net> wrote:
> On Oct 12, 6:41?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:

>> So, I'm browsing a program and I see
>>
>>   : blather   ... ?>bar ?... ;
>>
>> and then I type
>>
>>   locate >bar
>>
>> and I see
>>
>>   : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;
>>
>> so I type
>>
>>   locate bar2
>>
>> I guess the next thing I'd see would be
>>
>>   bar2 ?
> 
> Why is locate bar2 doing that?  Because its not in the namespace?

Because it's not in the current search order.

> In the context of published libraries of source, if private words are
> not hidden, when there is a namespace conflict among supplementary
> words, wouldn't "locate bar2" give the most recently defined bar2,
> which may well not be the one being used above?

Indeed it would.

>> I'd swear profusely at the author, then try to figure out which module
>> contained BAR2 . If this happened more than once I'd figure out how
>> to disable the module system in order to make browsing easier.
> 
>> I suppose, in an ideal world, LOCATE >BAR would open the module FOO
>> (because >BAR is exported from FOO).  I'm not sure how this would work
>> in practice.
> 
> If you have included bar2 from a published library, you get to the
> point where you are better off opening the published library file and
> seeing what it says about what bar2 is doing.

I'm not sure what this means.  In a forth context, what does it mean
to "open the published library file" ?

> If bar2 was compiled into the open, and the private words are what is
> located in "the module", locate would find bar2.

I'm sorry, I don't understand what this means.  If BAR2 is private,
it'll be in a wordlist that's not in the default search order.

> If it also tells you which file it was compiled from, you can open
> that file if you want the nitty gritty on what is behind the public
> word.

Right: that's the normal situation, without words in separate
wordlists.

> Indeed, if there is a glossary of the public words included in the
> module, and you were unfamiliar with bar2, you probably have a set of
> words described there that you ought to know.
> 
> Now, you can backtrack one, look at the file that included >bar, and
> then work out from the documentation or guess which of the include
> files bar2 came from ... but IMHO, having the public word in the open
> is better in that circumstance, since you can go directly to that file
> that defined that word and its partners.

Of course.  The problem is browsing the private words.

Andrew.

On Oct 12, 10:19=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:

> > If you have included bar2 from a published library, you get to the
> > point where you are better off opening the published library file and
> > seeing what it says about what bar2 is doing.
>
> I'm not sure what this means. =A0In a forth context, what does it mean
> to "open the published library file" ?

You or the system has at some point included a published library file
that uses the modular programming wordset.

If locate is finding the definition for the word in the file it was
included from, it can tell you which file its in as well as show the
definition.

If all their public words are in the open, then its reasonably
straightforward to extend locate to accomodate the private words in a
module for the kind of process you stepped through ~ when you
successfully locate a word in the namespace, remember the pointer to
its include file, and if you fail to locate a word in the namespace,
"locate" tries a text search in the include file of the most recent
successfully located word.

If the public words are out of sight, its a lot harder to do that.

> > If bar2 was compiled into the open, and the private words are what is
> > located in "the module", locate would find bar2.

> I'm sorry, I don't understand what this means. =A0If BAR2 is private,
> it'll be in a wordlist that's not in the default search order.

Where you described being unable to "locate bar2", in the modular
programming approach that Krishna and David are suggesting, that could
well be a public word, since the "module" has all the public words in
their own distinct wordlist. So in that approach, "locate xyz" can
fail for xyz that is a public word.

In the modular programming approach I was suggesting, and which Ms.
Rather reports on Swiftforth having, the public words are compiled
into the original compilation wordlist when the file containing the
module is included. So in that approach, if "locate xyz" fails on a
public words, that's not the library's fault ~ it compiled the public
words where the includer told it to.

> > If it also tells you which file it was compiled from, you can open
> > that file if you want the nitty gritty on what is behind the public
> > word.

> Right: that's the normal situation, without words in separate
> wordlists.

The normal situation in the Forth Scientific Library is *already* to
have words in separate wordlists. Remember that this started from
disatisfaction with the lack of separation in PRIVATE words from
putting all of the FSL PRIVATE words in the *same* private wordlist.

What is the "normal" situation depends largely on what kind of
application you are coding. A structured wordspace may be more
appealing for a GUI Windows32 front end, or a GUI front end that uses
the same interface for Windows-Mobile, Linux-Android and iOS, and in
the FSL situation in particular, there's some functions in there I am
not terribly interested in having all of their innards in the main
Forth wordlist.

> > Indeed, if there is a glossary of the public words included in the
> > module, and you were unfamiliar with bar2, you probably have a set of
> > words described there that you ought to know.

> > Now, you can backtrack one, look at the file that included >bar, and
> > then work out from the documentation or guess which of the include
> > files bar2 came from ... but IMHO, having the public word in the open
> > is better in that circumstance, since you can go directly to that file
> > that defined that word and its partners.

> Of course. =A0The problem is browsing the private words.

If its a well designed library, you may not need to browse the private
words. And if you are going to work with the private words, in a
library module written by someone else, you really ought to open up
the file that contains the module and look for it in there, which is
something that ought to be straightforward to add to locate if the
public words are, as their name suggests, defined out in public.

For a more tightly integrated "locate", if the implementation allows
adding additional side information to definitions, then with the
layout of:
   Begin-Module: foo
   ...
   PUBLIC
   ...
   PRIVATE
   ...
   PUBLIC
   ...
   End-Module

.... PUBLIC could remember the "extra" wordlists on the search order
stack (since it can determine the original search order height and the
current search order height, and throw a toggle to include a pointer
to that information in definitions until PRIVATE. Then the "remember
the last successful locate" could go look in those wordlists if
failing to find a word in the visible dictionary.

Working on an application in a small, memory constrained chip makes it
more likely that integrating a single namespace by hand is simpler
than using library source as-is ~ in which case using a flat version
of the modular programming tools is fine:

: PUBLIC ( -- ) ;
: PRIVATE ( -- ) ;
: EXPORT ( -- ) ;
: End-module ( -- ) ;
: Toolkit: ( "name" -- ) CREATE DOES> DROP ;
: Begin-module: ( "name" -- ) Toolkit: ;
: Open-toolkit: ( "name" -- )  PARSE-NAME 2DROP ;
: Open-export: ( "name" -- ) PARSE-NAME 2DROP ;

Working on an application that has a complex enough namespace, or
includes enough external library source written by some other author
with their own approach to naming, to make a structured namespace
useful implies that the benefit of being able to discriminate between
more widely and more narrowly visible word has a substantial benefit
to offer.

The challenge, then, is how to do that for the best net benefit, by
delivering those benefits at the lowest additional cost, including
things such as the extra complexity of browsing a structured namespace
versus an unstructured namespace.

BruceMcF <agila61@netscape.net> wrote:
> On Oct 12, 10:19?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
> 
>> > If you have included bar2 from a published library, you get to the
>> > point where you are better off opening the published library file and
>> > seeing what it says about what bar2 is doing.
>>
>> I'm not sure what this means. ?In a forth context, what does it mean
>> to "open the published library file" ?
> 
> You or the system has at some point included a published library file
> that uses the modular programming wordset.
> 
> If locate is finding the definition for the word in the file it was
> included from, it can tell you which file its in as well as show the
> definition.

Sure, it can, but what for?  I just want to see the source for that
word.

> If all their public words are in the open, then its reasonably
> straightforward to extend locate to accomodate the private words in a
> module for the kind of process you stepped through ~ when you
> successfully locate a word in the namespace, remember the pointer to
> its include file, and if you fail to locate a word in the namespace,
> "locate" tries a text search in the include file of the most recent
> successfully located word.

Eww.  I guess it would work, but that really is a nasty kludge.

> If the public words are out of sight, its a lot harder to do that.
> 
>> > If bar2 was compiled into the open, and the private words are what is
>> > located in "the module", locate would find bar2.
> 
>> I'm sorry, I don't understand what this means. If BAR2 is private,
>> it'll be in a wordlist that's not in the default search order.
> 
> Where you described being unable to "locate bar2", in the modular
> programming approach that Krishna and David are suggesting, that could
> well be a public word, since the "module" has all the public words in
> their own distinct wordlist. So in that approach, "locate xyz" can
> fail for xyz that is a public word.

But it's not public.  The whole point of the example is what happens
when you have a word

  : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;

where BAR2 is private.

> In the modular programming approach I was suggesting, and which Ms.
> Rather reports on Swiftforth having, the public words are compiled
> into the original compilation wordlist when the file containing the
> module is included. So in that approach, if "locate xyz" fails on a
> public words, that's not the library's fault ~ it compiled the public
> words where the includer told it to.

Why would it fail on a public word?  We're not talking about it
failing on a public word.

>> > Indeed, if there is a glossary of the public words included in the
>> > module, and you were unfamiliar with bar2, you probably have a set of
>> > words described there that you ought to know.
> 
>> > Now, you can backtrack one, look at the file that included >bar, and
>> > then work out from the documentation or guess which of the include
>> > files bar2 came from ... but IMHO, having the public word in the open
>> > is better in that circumstance, since you can go directly to that file
>> > that defined that word and its partners.
> 
>> Of course. The problem is browsing the private words.
> 
> If its a well designed library, you may not need to browse the
> private words.

Possibly.  But if a library is going to survive, it'll be improved,
extended, and so on.  And Forth is very much not about "don't worry,
nothing to see."  (IMO!)
 
> And if you are going to work with the private words, in a library
> module written by someone else, you really ought to open up the file
> that contains the module and look for it in there, which is
> something that ought to be straightforward to add to locate if the
> public words are, as their name suggests, defined out in public.

I'm assuming that when you're programming in Forth, you want to use
your usual Forth toolset.  That you don't want to use some different
toolset to browse just because the author of a library decided to use
"modular" Forth.

> For a more tightly integrated "locate", if the implementation allows
> adding additional side information to definitions, then with the
> layout of:
>   Begin-Module: foo
>   ...
>   PUBLIC
>   ...
>   PRIVATE
>   ...
>   PUBLIC
>   ...
>   End-Module
> 
> ... PUBLIC could remember the "extra" wordlists on the search order
> stack (since it can determine the original search order height and the
> current search order height, and throw a toggle to include a pointer
> to that information in definitions until PRIVATE. Then the "remember
> the last successful locate" could go look in those wordlists if
> failing to find a word in the visible dictionary.

That is starting to should like something that might work.

> Working on an application that has a complex enough namespace, or
> includes enough external library source written by some other author
> with their own approach to naming, to make a structured namespace
> useful implies that the benefit of being able to discriminate
> between more widely and more narrowly visible word has a substantial
> benefit to offer.

Sure.  It makes sense to do this when there is a clash and there isn't
much of a downside.

> The challenge, then, is how to do that for the best net benefit, by
> delivering those benefits at the lowest additional cost, including
> things such as the extra complexity of browsing a structured
> namespace versus an unstructured namespace.

Indeed.  It seems to me that the whole issue of how one would work on
a system consisting of these modules has hitherto been ignored.
Either the people working on such systems just put up with the Forth
tools like LOCATE no longer working or they never learned to use them
in the first place.  "LOCATE doesn't work.  Never mind, you can use
grep."  :-)

Andrew.

On Oct 12, 12:34=A0pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:

> > If locate is finding the definition for the word in the file it was
> > included from, it can tell you which file its in as well as show the
> > definition.

> Sure, it can, but what for? =A0I just want to see the source for that
> word.

When looking at a set of words written by someone else, its often
useful to step back and see how the whole system fits together.

> > If all their public words are in the open, then its reasonably
> > straightforward to extend locate to accomodate the private words in a
> > module for the kind of process you stepped through ~ when you
> > successfully locate a word in the namespace, remember the pointer to
> > its include file, and if you fail to locate a word in the namespace,
> > "locate" tries a text search in the include file of the most recent
> > successfully located word.

> Eww. =A0I guess it would work, but that really is a nasty kludge.

Why? If it works, it works, and you are where it does you the most
good to be if you are exploring a public word included from a library
file written by someone else.

> But it's not public. =A0The whole point of the example is what happens
> when you have a word

> =A0 : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;

> where BAR2 is private.

BUT when Krishna and David say "PUBLIC", its inside a wordlist defined
by the included file, accessed via the name of the module. What I'm
saying is that in their approach the *public* words are normally not
in your main wordlist, unless you cross compiled them into your main,
as in:

ONLY FORTH DEFINITION ALSO FOOBAR
: foo foo ;
: bar bar ;

....

And, BTW, consider what *that* does to exploring code with locate.

> > In the modular programming approach I was suggesting, and which Ms.
> > Rather reports on Swiftforth having, the public words are compiled
> > into the original compilation wordlist when the file containing the
> > module is included. So in that approach, if "locate xyz" fails on a
> > public words, that's not the library's fault ~ it compiled the public
> > words where the includer told it to.

> Why would it fail on a public word? =A0We're not talking about it
> failing on a public word.

If you restart the search order, all of the public words in all of the
modules that are defined using the "named public wordlist" approach
are *inside* a wordlist in the original compilation directory.

> > If its a well designed library, you may not need to browse the
> > private words.

> Possibly. =A0But if a library is going to survive, it'll be improved,
> extended, and so on. =A0And Forth is very much not about "don't worry,
> nothing to see." =A0(IMO!)

Yes, but the problem you originally set out involved browsing code
that is a *user* of the library. With the "named private" modular
wordlist approach, if you are developing the library, you only have to
bring the private wordlist into the search order and locate works just
fine.

> > And if you are going to work with the private words, in a library
> > module written by someone else, you really ought to open up the file
> > that contains the module and look for it in there, which is
> > something that ought to be straightforward to add to locate if the
> > public words are, as their name suggests, defined out in public.

> I'm assuming that when you're programming in Forth, you want to use
> your usual Forth toolset. =A0That you don't want to use some different
> toolset to browse just because the author of a library decided to use
> "modular" Forth.

Which gets right back to problem domain. If your usual problem domain
is one where a structured wordspace is a useful thing, your usual
Forth toolset won't come a cropper when faced with a structured
wordspace.

> > For a more tightly integrated "locate", if the implementation allows
> > adding additional side information to definitions, then with the
> > layout of:
> > =A0 Begin-Module: foo
> > =A0 ...
> > =A0 PUBLIC
> > =A0 ...
> > =A0 PRIVATE
> > =A0 ...
> > =A0 PUBLIC
> > =A0 ...
> > =A0 End-Module
>
> > ... PUBLIC could remember the "extra" wordlists on the search order
> > stack (since it can determine the original search order height and the
> > current search order height, and throw a toggle to include a pointer
> > to that information in definitions until PRIVATE. Then the "remember
> > the last successful locate" could go look in those wordlists if
> > failing to find a word in the visible dictionary.

> That is starting to should like something that might work.

It does imply that well-structured modules do not muck about with the
search order while compiling words to PUBLIC ~ but I believe that well-
structured modules should not much about with the search order while
compiling words to PUBLIC, so I'm fine with that.

If your Forth development environment has a good Forth source file
viewer, with reload, load block, and load through to mark, and you
primarily access the dictionary browser via the Forth source file
viewer, dictionary locate by right clicking the word and picking that
option inside the Forth source file viewer is nice ~ worth adding to
the right click menu if its not there already.

If you are doing a dictionary browse of a public word used in one file
but included from another, and then when looking in it, you click on a
word that is not in the search order, but is located in the file that
the word you are looking at was included from ... popping up a second
file viewer window to look at where that word is located in the file
would seem to be a quite natural extension. It doesn't need to be all
that smart ~ hitting the search again control key or function key once
or twice will bring up the definition if the first hit is in the
descriptive documentation.

> > Working on an application that has a complex enough namespace, or
> > includes enough external library source written by some other author
> > with their own approach to naming, to make a structured namespace
> > useful implies that the benefit of being able to discriminate
> > between more widely and more narrowly visible word has a substantial
> > benefit to offer.

> Sure. =A0It makes sense to do this when there is a clash and there isn't
> much of a downside.

When you have created the structure, application side, then presumably
you know how to put the namespace into the appropriate state for the
browsing you want to do.

The issue you raise is what about when the structure is created
library side, and the structure *blindsides* the library user.

(1) That's a *more* serious problem if even the public words are
placed into their own wordlist defined inside the external library
code.

(2) If its *just* the private words placed into their own wordlist,
version of the modular development tools tailored to a locate user
could form a linked list of all the private wordlists they create, and
provide a new version of locate that steps through each of those in
turn if the word you are trying to locate is outside the public
namespace.

> > The challenge, then, is how to do that for the best net benefit, by
> > delivering those benefits at the lowest additional cost, including
> > things such as the extra complexity of browsing a structured
> > namespace versus an unstructured namespace.

> Indeed. =A0It seems to me that the whole issue of how one would work on
> a system consisting of these modules has hitherto been ignored.
> Either the people working on such systems just put up with the Forth
> tools like LOCATE no longer working or they never learned to use them
> in the first place. =A0"LOCATE doesn't work. =A0Never mind, you can use
> grep." =A0:-)

Or their Forth tools don't have any trouble with that.

On 10/10/2011 17:13, Krishna Myneni wrote:
> On Oct 10, 10:04 am, Gerry Jackson<ge...@jackson9000.fsnet.co.uk>
> wrote:
[...]

>> I've been following the discussion with some interest and, while I have
>> sort of lost track of where different people stand on the topic, it
>> seems to me that a consensus is not emerging. ...
>
> Yes, I also do not see a clear consensus emerging from this
> discussion.
>
>> ... I am increasingly feeling
>> that it might be better to split responsibilities for modularisation of
>> code between the modules and the user.
>>
>> For example if there were 4 words available for use by a module:
>>
>> BEGIN-MODULE which saves the compilation wordlist and top wordlist in
>> the search order as the public and private wordlists respectively,
>> setting the private wordlist as the compilation wordlist.
>>
>> PRIVATE and PUBLIC setting their wordlists as the compilation wordlist.
>>
>> END-MODULE restores the original compilation wordlist
>>
>> Then it would be up to the user to decide whether to name a module or
>> not, whether to create a new public wordlist or not, whether to have a
>> private wordlist (use the public one for both if a flat namespace is
>> wanted) and so on.
>>
>> The user sets the strategy, the module does the mechanics.
>>
>
> That's certainly a partial solution, but really only useful for
> anonymous modules. It means that a module cannot declare its
> dependencies on other modules -- one of the benefits provided by named
> modules. The user of the modules has to set up dependencies for the
> search order prior to loading each module, and then removing the
> dependencies from the search order. This could become extremely
> tedious. In contrast, DNW's root-module.fs, and my module.fs, provide
> a very convenient way of automatically bringing dependencies into the
> search order through the module names, and automatically removing them
> from the search order.
>

Andrew raised the same point. I was envisaging that the user 
(application side to use Bruce's terminology) would define a word 
called, say, MODULE  ( "name" -- ) that is available for the user and 
modules to use. When executed MODULE would set the compilation wordlist 
and top of the search order according to the user's requirements before 
INCLUDEing a library module. The library module would be able to use 
MODULE when including another module it depended on.

I need to think this through but haven't much time at present unfortunately.

-- 
Gerry

Where do the hidden words go? From the point of view of the adapter, surely they must remain visible somewhere.

Cheers Jacko

On Oct 12, 3:05=A0pm, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
wrote:
> On 10/10/2011 17:13, Krishna Myneni wrote:> On Oct 10, 10:04 am, Gerry Ja=
ckson<ge...@jackson9000.fsnet.co.uk>
> > wrote:
>
> [...]
>
>
>
> >> I've been following the discussion with some interest and, while I hav=
e
> >> sort of lost track of where different people stand on the topic, it
> >> seems to me that a consensus is not emerging. ...
>
> > Yes, I also do not see a clear consensus emerging from this
> > discussion.
>
> >> ... I am increasingly feeling
> >> that it might be better to split responsibilities for modularisation o=
f
> >> code between the modules and the user.
>
> >> For example if there were 4 words available for use by a module:
>
> >> BEGIN-MODULE which saves the compilation wordlist and top wordlist in
> >> the search order as the public and private wordlists respectively,
> >> setting the private wordlist as the compilation wordlist.
>
> >> PRIVATE and PUBLIC setting their wordlists as the compilation wordlist=
..
>
> >> END-MODULE restores the original compilation wordlist
>
> >> Then it would be up to the user to decide whether to name a module or
> >> not, whether to create a new public wordlist or not, whether to have a
> >> private wordlist (use the public one for both if a flat namespace is
> >> wanted) and so on.
>
> >> The user sets the strategy, the module does the mechanics.
>
> > That's certainly a partial solution, but really only useful for
> > anonymous modules. It means that a module cannot declare its
> > dependencies on other modules -- one of the benefits provided by named
> > modules. The user of the modules has to set up dependencies for the
> > search order prior to loading each module, and then removing the
> > dependencies from the search order. This could become extremely
> > tedious. In contrast, DNW's root-module.fs, and my module.fs, provide
> > a very convenient way of automatically bringing dependencies into the
> > search order through the module names, and automatically removing them
> > from the search order.
>
> Andrew raised the same point. I was envisaging that the user
> (application side to use Bruce's terminology) would define a word
> called, say, MODULE =A0( "name" -- ) that is available for the user and
> modules to use. When executed MODULE would set the compilation wordlist
> and top of the search order according to the user's requirements before
> INCLUDEing a library module. The library module would be able to use
> MODULE when including another module it depended on.
>
> I need to think this through but haven't much time at present unfortunate=
ly.
>
> --
> Gerry

Hi Gerry,

In my implementation, one should be able to do the following from the
"user" side:

MODULE: <name>
<mod_a> <mod_b>   \ dependencies
include modc.fs

In the above example, modc.fs will start with BEGIN-MODULE  and end
with END-MODULE , with PUBLIC: and PRIVATE: sections in between. In
this way, the module naming and setting up the dependencies is done on
the "user" side. However, the only drawback (from my perspective) is
that the search order is not restored to the state prior to the
dependencies line. One would have to do a PREVIOUS for each declared
dependency to return to the prior search order. Also, note that in my
implementation, MODULE: does not modify the search order or
compilation wordlist. It creates a named reference for the new module,
which goes into the compilation wordlist prior to the MODULE:
statement.

Krishna

BruceMcF <agila61@netscape.net> wrote:
> On Oct 12, 12:34?pm, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
> 
>> > If locate is finding the definition for the word in the file it was
>> > included from, it can tell you which file its in as well as show the
>> > definition.
> 
>> Sure, it can, but what for? ?I just want to see the source for that
>> word.
> 
> When looking at a set of words written by someone else, its often
> useful to step back and see how the whole system fits together.

Of course, but you shouldn't have to.

>> > If all their public words are in the open, then its reasonably
>> > straightforward to extend locate to accomodate the private words in a
>> > module for the kind of process you stepped through ~ when you
>> > successfully locate a word in the namespace, remember the pointer to
>> > its include file, and if you fail to locate a word in the namespace,
>> > "locate" tries a text search in the include file of the most recent
>> > successfully located word.
> 
>> Eww. I guess it would work, but that really is a nasty kludge.
> 
> Why? If it works, it works, and you are where it does you the most
> good to be if you are exploring a public word included from a library
> file written by someone else.

I don't think this is so.  It forces you away from the interactive
exploratory environment that is one of Forth's greatest strengths into
the file editor environment.

>> But it's not public. ?The whole point of the example is what happens
>> when you have a word
> 
>>   : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;
> 
>> where BAR2 is private.
> 
> BUT when Krishna and David say "PUBLIC", its inside a wordlist defined
> by the included file, accessed via the name of the module. What I'm
> saying is that in their approach the *public* words are normally not
> in your main wordlist, unless you cross compiled them into your main,
> as in:
> 
> ONLY FORTH DEFINITION ALSO FOOBAR
> : foo foo ;
> : bar bar ;
> 
> ...
> 
> And, BTW, consider what *that* does to exploring code with locate.

It's a complete disaster, obviously.
 
>> > If its a well designed library, you may not need to browse the
>> > private words.
> 
>> Possibly. But if a library is going to survive, it'll be improved,
>> extended, and so on. And Forth is very much not about "don't worry,
>> nothing to see." (IMO!)
> 
> Yes, but the problem you originally set out involved browsing code
> that is a *user* of the library.

When browsing, you look at Word X, which calls Word Y, which calls Word
Z.  You don't stop browsing at module boundaries or at private/public
boundaries.

> With the "named private" modular wordlist approach, if you are
> developing the library, you only have to bring the private wordlist
> into the search order and locate works just fine.

Sheesh.  How do you even know what the private wordlist is called?
You're just browsing code.  You don't want to have to search the files
in which a word is defined to find out which private wordlist it's in.

>> > And if you are going to work with the private words, in a library
>> > module written by someone else, you really ought to open up the file
>> > that contains the module and look for it in there, which is
>> > something that ought to be straightforward to add to locate if the
>> > public words are, as their name suggests, defined out in public.
> 
>> I'm assuming that when you're programming in Forth, you want to use
>> your usual Forth toolset. ?That you don't want to use some different
>> toolset to browse just because the author of a library decided to use
>> "modular" Forth.
> 
> Which gets right back to problem domain. If your usual problem domain
> is one where a structured wordspace is a useful thing, your usual
> Forth toolset won't come a cropper when faced with a structured
> wordspace.

OK, let's examine that more closely. We have a library that might be
useful in some circumstances. We have a bunch of Forths and Forth
programmers.

As a user, I might want to use some of this library code.  But why
should I have a special toolset?  Is it your claim that FSL is only
useful on Forths whose usual problem domain is one where a structured
wordspace such a useful thing that special tools have been developed
to cope with it?  If so, I suspect FSL's potential user base is very
small.

>> > current search order height, and throw a toggle to include a pointer
>> > to that information in definitions until PRIVATE. Then the "remember
>> > the last successful locate" could go look in those wordlists if
>> > failing to find a word in the visible dictionary.
> 
>> That is starting to should like something that might work.
> 
> It does imply that well-structured modules do not muck about with the
> search order while compiling words to PUBLIC ~ but I believe that well-
> structured modules should not much about with the search order while
> compiling words to PUBLIC, so I'm fine with that.
> 
> If your Forth development environment has a good Forth source file
> viewer, with reload, load block, and load through to mark, and you
> primarily access the dictionary browser via the Forth source file
> viewer, dictionary locate by right clicking the word and picking that
> option inside the Forth source file viewer is nice ~ worth adding to
> the right click menu if its not there already.

I suppose so, but I guess I don't know why anyone would want such a
thing.  LOCATE is a simple Forth tool that does the job.

> If you are doing a dictionary browse of a public word used in one file
> but included from another, and then when looking in it, you click on a
> word that is not in the search order, but is located in the file that
> the word you are looking at was included from ... popping up a second
> file viewer window to look at where that word is located in the file
> would seem to be a quite natural extension. It doesn't need to be all
> that smart ~ hitting the search again control key or function key once
> or twice will bring up the definition if the first hit is in the
> descriptive documentation.

Sonds cute but rather baroque, and it takes you away from the keyboard
interprter, which IME is always a distraction.

>> > Working on an application that has a complex enough namespace, or
>> > includes enough external library source written by some other author
>> > with their own approach to naming, to make a structured namespace
>> > useful implies that the benefit of being able to discriminate
>> > between more widely and more narrowly visible word has a substantial
>> > benefit to offer.
> 
>> Sure. It makes sense to do this when there is a clash and there isn't
>> much of a downside.
> 
> When you have created the structure, application side, then presumably
> you know how to put the namespace into the appropriate state for the
> browsing you want to do.
> 
> The issue you raise is what about when the structure is created
> library side, and the structure *blindsides* the library user.

Yes.

> (1) That's a *more* serious problem if even the public words are
> placed into their own wordlist defined inside the external library
> code.

Yes.  The more intrusive this stuff is, the worse for the library
user.

> (2) If its *just* the private words placed into their own wordlist,
> version of the modular development tools tailored to a locate user
> could form a linked list of all the private wordlists they create, and
> provide a new version of locate that steps through each of those in
> turn if the word you are trying to locate is outside the public
> namespace.

Right.  So, there is a way to create something that might work.

>> > The challenge, then, is how to do that for the best net benefit, by
>> > delivering those benefits at the lowest additional cost, including
>> > things such as the extra complexity of browsing a structured
>> > namespace versus an unstructured namespace.
> 
>> Indeed. It seems to me that the whole issue of how one would work on
>> a system consisting of these modules has hitherto been ignored.
>> Either the people working on such systems just put up with the Forth
>> tools like LOCATE no longer working or they never learned to use them
>> in the first place. "LOCATE doesn't work. Never mind, you can use
>> grep." :-)
> 
> Or their Forth tools don't have any trouble with that.

Maybe, but I doubt it.  I think it's just been ignored.

Andrew.

On Oct 13, 5:37=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> BruceMcF <agil...@netscape.net> wrote:
> > On Oct 12, 12:34?pm, Andrew Haley <andre...@littlepinkcloud.invalid>
> > wrote:

> >> > If locate is finding the definition for the word in the file
> >> > it was included from, it can tell you which file its in as
> >> > well as show the definition.

> >> Sure, it can, but what for? ?I just want to see the source for that
> >> word.

> > When looking at a set of words written by someone else, its often
> > useful to step back and see how the whole system fits together.

> Of course, but you shouldn't have to.

That's a design target for one style of development tool.

> >> > If all their public words are in the open, then its
> >> > reasonably straightforward to extend locate to accomodate
> >> > the private words in a module for the kind of process you
> >> > stepped through ~ when you successfully locate a word in
> >> > the namespace, remember the pointer to its include file, and
> >> > if you fail to locate a word in the namespace, "locate" tries
> >> > a text search in the include file of the most recent
> >> > successfully located word.

> >> Eww. I guess it would work, but that really is a nasty kludge.

> > Why? If it works, it works, and you are where it does you the most
> > good to be if you are exploring a public word included from a library
> > file written by someone else.

> I don't think this is so. =A0It forces you away from the interactive
> exploratory environment that is one of Forth's greatest strengths into
> the file editor environment.

If it was designed in a way that forced you away from the interactive
exploratory environment, I'd say it was implemented badly.

> >> But it's not public. ?The whole point of the example is what happens
> >> when you have a word

> >> =A0 : >bar ( x1 x2 -- ) bar2 ! bar1 ! ;

> >> where BAR2 is private.

> > BUT when Krishna and David say "PUBLIC", its inside a wordlist
> > defined by the included file, accessed via the name of the module.
> > What I'm saying is that in their approach the *public* words are
> > normally not in your main wordlist, unless you cross compiled them
> > into your main, as in:

> > ONLY FORTH DEFINITION ALSO FOOBAR
> > : foo foo ;
> > : bar bar ;

> > ...
>
> > And, BTW, consider what *that* does to exploring code with locate.

> It's a complete disaster, obviously.

So in addition to not wanting to have published library source using
PRIVATE and PUBLIC at all, can I take it that you would prefer any
library source that did so to use a system that compiled PUBLIC code
into the original compilation wordlist?


> >> > If its a well designed library, you may not need to browse the
> >> > private words.

> >> Possibly. But if a library is going to survive, it'll be improved,
> >> extended, and so on. And Forth is very much not about "don't worry,
> >> nothing to see." (IMO!)

> > Yes, but the problem you originally set out involved browsing code
> > that is a *user* of the library.

> When browsing, you look at Word X, which calls Word Y, which calls Word
> Z. =A0You don't stop browsing at module boundaries or at private/public
> boundaries.

But your argument above seems stronger than that: not just that the
private/public boundary should not be a barrier to browsing, but that
the private/public should be invisible when browsing.

I can see the first, but I'm a bit skeptical about the latter. For one
thing, even when you want to browse beyond the boundary, you are now
entering a distinct system with its own mindset, a defined glossary of
public words, and hopefully a set of tests of the public words
compliance with their specified behavior. So you are browsing a
distinctive landscape with its own distinctive features, and that
context is useful information while browsing.

And of course, in a sufficiently large total namespace, not everyone
*will* want to browse beyond the public/private boundary of every
module ~ and if the boundary is invisible, they'll find themselves
browsing into territory they wouldn't have explored in that way if
they'd known.

Now without any substantial published library source, we've got a
development process supported on a combination of the implementation
side and the development environment side to explore the application
side. With a substantial published library source, we have the same
combination to explore both the application side and the library side.

You've got a strong preference for a seamless crossing of the
application side / library side boundary. I'm not sure I mind a seam,
if its not much trouble crossing it.

But with respect to the seamless crossing, I'd argue that its easier
to achieve if the application side / library side boundary is between
public and private than if the application side / library side
boundary is between the entire set of words defined by the library
module and the including application.

> > With the "named private" modular wordlist approach, if you are
> > developing the library, you only have to bring the private wordlist
> > into the search order and locate works just fine.

> Sheesh. =A0How do you even know what the private wordlist is called?
> You're just browsing code. =A0You don't want to have to search the files
> in which a word is defined to find out which private wordlist it's in.

If you are working *on* that module in the library, then you probably
want to know the name of the module independently of whether or not
you are browsing an instance of the compiled source.

In any event, you're imagining that its not *been* made easy to do so.
My concern with that is whether it *could be* made easy to do so.
Since the file can easily be run through a filter that says what
modules are defined in it, and what private and public words are
defined in it, then so long as the semantics are adequate to account
for all words it defines *anywhere*, it *could be* made easy to do.

Since that can all be automated, and if an instance of the source has
been compiled, those names can be automatically tied to actual
wordlists in the dictionary, all of the "wait a minute, where IS that
word, anyway?" can certainly be made to go away if you know the file
its in, know that its a library module, and can bring that kind of
tool to bear.

This, however, does imply that the library module system in question
should *only* use a defined set of words to manipulate the search
order. I think I'd advise that in any event, because of the way it
supports using implementation provided VOCABULARY despite the fact
that different implementations do VOCABULARY in different ways.

In particular, an extension of "locate" that can make use of that
information would be straightforward, without requiring any
specialized extensions to either the implementation or the reference
library module wordset.

> OK, let's examine that more closely. We have a library that might be
> useful in some circumstances. We have a bunch of Forths and Forth
> programmers.

> As a user, I might want to use some of this library code. =A0But why
> should I have a special toolset?

Why do we ever want tools designed for the job when we are working on
a specific task? Because working with tools designed for the task make
it easier to do the job.

We are talking about a library approach in particular that has an
implementation specific utility file that is loaded, and then portable
library modules.

>=A0Is it your claim that FSL is only
> useful on Forths whose usual problem domain is one where a structured
> wordspace such a useful thing that special tools have been developed
> to cope with it? =A0If so, I suspect FSL's potential user base is very
> small.

There's not much of an issue with FSL at present, since its PRIVATE
wordlist is a common pool, so you don't have to know which module a
private word is in in order to make sure that all private words are in
the search order ~ simply bring the single private wordlist into the
search order, and they're all visible.

And given the nature of the library, lots of users will be content
with knowing what the public words *do*, without needing to inquire
how they do it.

However, if you want to go browsing in the way you describe, sometimes
you run into the wrong private word, since name clashes means that its
the even of a name clash that its the private word in the most
recently compiled module that is visible, which might not be the
module you are browsing.


> > If your Forth development environment has a good Forth source file
> > viewer, with reload, load block, and load through to mark, and you
> > primarily access the dictionary browser via the Forth source file
> > viewer, dictionary locate by right clicking the word and picking that
> > option inside the Forth source file viewer is nice ~ worth adding to
> > the right click menu if its not there already.

> I suppose so, but I guess I don't know why anyone would want such a
> thing. =A0LOCATE is a simple Forth tool that does the job.

To avoid the annoyance of *having to* retype the word that you are
already looking at. The command line interpreter is a valuable fall
back to have, but I'd not want to be forced to use it as the primary
dictionary browser when working in an environment where PAGE and AT-XY
work.

Or, IOW, the same reason I have always preferred a screen editor to a
line editor.

> > If you are doing a dictionary browse of a public word used
> > in one file but included from another, and then when looking
> > in it, you click on a word that is not in the search order,
> > but is located in the file that the word you are looking at
> > was included from ... popping up a second file viewer window
> > to look at where that word is located in the file would seem
> > to be a quite natural extension. It doesn't need to be all that
> > smart ~ hitting the search again control key or function key once
> > or twice will bring up the definition if the first hit is in the
> > descriptive documentation.

> Sounds cute but rather baroque, and it takes you away from the keyboard
> interpreter, which IME is always a distraction.

Yes, escaping the constraints and "looking at the source through a
mailbox slit" of the command line environment is the point of it,
which is why in Forth, screen based development environments have been
developed in addition to command line oriented development
environments for over forty years.

> >> Indeed. It seems to me that the whole issue of how one would
> >> work on a system consisting of these modules has hitherto been
> >> ignored. Either the people working on such systems just put up
> >> with the Forth tools like LOCATE no longer working or they never
> >> learned to use them in the first place. "LOCATE doesn't work.
> >> Never mind, you can use grep." :-)
>
> > Or their Forth tools don't have any trouble with that.
>
> Maybe, but I doubt it. =A0I think it's just been ignored.

For people who develop by shuttling back and forth between the command
line and the integrated source file editor, or shuttling back and
forth between their preferred general program editor and newly created
instance of Forth, they might not do much dictionary browsing ~ and
then, mostly to squash unanticipated name-clash bugs, where the
PRIVATE/PUBLIC approach already tends to prevent those bugs from
occurring in the first place.

BruceMcF <agila61@netscape.net> wrote:
> On Oct 13, 5:37?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>> BruceMcF <agil...@netscape.net> wrote:
>> > On Oct 12, 12:34?pm, Andrew Haley <andre...@littlepinkcloud.invalid>
>> > wrote:
> 
> So in addition to not wanting to have published library source using
> PRIVATE and PUBLIC at all, can I take it that you would prefer any
> library source that did so to use a system that compiled PUBLIC code
> into the original compilation wordlist?

Absolutely.  I think I said that already.

>> >> > If its a well designed library, you may not need to browse the
>> >> > private words.
> 
>> >> Possibly. But if a library is going to survive, it'll be improved,
>> >> extended, and so on. And Forth is very much not about "don't worry,
>> >> nothing to see." (IMO!)
> 
>> > Yes, but the problem you originally set out involved browsing code
>> > that is a *user* of the library.
> 
>> When browsing, you look at Word X, which calls Word Y, which calls Word
>> Z. You don't stop browsing at module boundaries or at private/public
>> boundaries.
> 
> But your argument above seems stronger than that: not just that the
> private/public boundary should not be a barrier to browsing, but that
> the private/public should be invisible when browsing.
> 
> I can see the first, but I'm a bit skeptical about the latter.

Well, invisible is ideal.  Anything that forces you to have to do
extra work beyond thinking about your program itself, is, obviously,
part of the downside.

> For one thing, even when you want to browse beyond the boundary, you
> are now entering a distinct system with its own mindset, a defined
> glossary of public words, and hopefully a set of tests of the public
> words compliance with their specified behavior. So you are browsing
> a distinctive landscape with its own distinctive features, and that
> context is useful information while browsing.
>
> And of course, in a sufficiently large total namespace, not everyone
> *will* want to browse beyond the public/private boundary of every
> module ~ and if the boundary is invisible, they'll find themselves
> browsing into territory they wouldn't have explored in that way if
> they'd known.

Perhaps so.  I rather appreciate the (non-Forth) tools I use all the
time that quite happily let you e.g. step into "system" code.  It
doesn't work for the Linux kernel, which really is the other side of a
wall, but I'd consider that a bug.
 
> Now without any substantial published library source, we've got a
> development process supported on a combination of the implementation
> side and the development environment side to explore the application
> side. With a substantial published library source, we have the same
> combination to explore both the application side and the library side.
> 
> You've got a strong preference for a seamless crossing of the
> application side / library side boundary. I'm not sure I mind a seam,
> if its not much trouble crossing it.

That's not so bad.

> But with respect to the seamless crossing, I'd argue that its easier
> to achieve if the application side / library side boundary is between
> public and private than if the application side / library side
> boundary is between the entire set of words defined by the library
> module and the including application.
> 
>> > With the "named private" modular wordlist approach, if you are
>> > developing the library, you only have to bring the private wordlist
>> > into the search order and locate works just fine.
> 
>> Sheesh. How do you even know what the private wordlist is called?
>> You're just browsing code. You don't want to have to search the files
>> in which a word is defined to find out which private wordlist it's in.
> 
> If you are working *on* that module in the library, then you probably
> want to know the name of the module independently of whether or not
> you are browsing an instance of the compiled source.

I doubt that very much.  Say I'm working on Module X, and Word Y
doesn't seem to do the right thing.  The next thing I want to do is
look at Word Y.  Do I want to know the name of the module in which Y
is held before I even look at Y?

> In any event, you're imagining that its not *been* made easy to do
> so.

I don't think I'm imagining anything.  I'm talking about the world as
it is, right now, if you load words defined using such modules into a
Forth system.

> My concern with that is whether it *could be* made easy to do so.
> Since the file can easily be run through a filter that says what
> modules are defined in it, and what private and public words are
> defined in it, then so long as the semantics are adequate to account
> for all words it defines *anywhere*, it *could be* made easy to do.

I agree, it could be done, with a new set of tools.

>> OK, let's examine that more closely. We have a library that might be
>> useful in some circumstances. We have a bunch of Forths and Forth
>> programmers.
> 
>> As a user, I might want to use some of this library code. But why
>> should I have a special toolset?
> 
> Why do we ever want tools designed for the job when we are working
> on a specific task? Because working with tools designed for the task
> make it easier to do the job.

But as a *user* of the FSL you probably just want the FSL.  You don't
necessarily want a bunch of new tools just because the designers of
the FSL like to use modules.

> We are talking about a library approach in particular that has an
> implementation specific utility file that is loaded, and then portable
> library modules.
> 
>>?Is it your claim that FSL is only
>> useful on Forths whose usual problem domain is one where a structured
>> wordspace such a useful thing that special tools have been developed
>> to cope with it? ?If so, I suspect FSL's potential user base is very
>> small.
> 
> There's not much of an issue with FSL at present, since its PRIVATE
> wordlist is a common pool, so you don't have to know which module a
> private word is in in order to make sure that all private words are in
> the search order ~ simply bring the single private wordlist into the
> search order, and they're all visible.

Good!

>> > If you are doing a dictionary browse of a public word used
>> > in one file but included from another, and then when looking
>> > in it, you click on a word that is not in the search order,
>> > but is located in the file that the word you are looking at
>> > was included from ... popping up a second file viewer window
>> > to look at where that word is located in the file would seem
>> > to be a quite natural extension. It doesn't need to be all that
>> > smart ~ hitting the search again control key or function key once
>> > or twice will bring up the definition if the first hit is in the
>> > descriptive documentation.
> 
>> Sounds cute but rather baroque, and it takes you away from the keyboard
>> interpreter, which IME is always a distraction.
> 
> Yes, escaping the constraints and "looking at the source through a
> mailbox slit" of the command line environment is the point of it,
> which is why in Forth, screen based development environments have been
> developed in addition to command line oriented development
> environments for over forty years.

I don't think this is anything to do with preferring "screen based
development environments" to the command line, etc.  Sometimes one is
appropriate, sometimes the other.  It's to do with *taking away* some
capabilities of the command line.

>> >> Indeed. It seems to me that the whole issue of how one would
>> >> work on a system consisting of these modules has hitherto been
>> >> ignored. Either the people working on such systems just put up
>> >> with the Forth tools like LOCATE no longer working or they never
>> >> learned to use them in the first place. "LOCATE doesn't work.
>> >> Never mind, you can use grep." :-)
>>
>> > Or their Forth tools don't have any trouble with that.
>>
>> Maybe, but I doubt it. ?I think it's just been ignored.
> 
> For people who develop by shuttling back and forth between the command
> line and the integrated source file editor, or shuttling back and
> forth between their preferred general program editor and newly created
> instance of Forth, they might not do much dictionary browsing ~ 

Bingo!  That's *exactly* my point.  They don't do it, so they don't
know that anyone else does it.  Unwittingly, they may destroy
something useful because they don't understand.  That's why I'm
spending all this time trying to explain.

> and then, mostly to squash unanticipated name-clash bugs, where the
> PRIVATE/PUBLIC approach already tends to prevent those bugs from
> occurring in the first place.

Is most of your development time fixing unanticipated name-clash bugs?
Really?

Andrew.

On 13/10/2011 02:03, Krishna Myneni wrote:

[...]

>
> In my implementation, one should be able to do the following from the
> "user" side:
>
> MODULE: <name>
> <mod_a>  <mod_b>    \ dependencies
> include modc.fs

We agree then.

>
> In the above example, modc.fs will start with BEGIN-MODULE  and end
> with END-MODULE , with PUBLIC: and PRIVATE: sections in between. In
> this way, the module naming and setting up the dependencies is done on
> the "user" side. However, the only drawback (from my perspective) is
> that the search order is not restored to the state prior to the
> dependencies line. One would have to do a PREVIOUS for each declared
> dependency to return to the prior search order. Also, note that in my
> implementation, MODULE: does not modify the search order or
> compilation wordlist. It creates a named reference for the new module,
> which goes into the compilation wordlist prior to the MODULE:
> statement.
>

If MODULE: doesn't change the search order (stated above) or record the 
search order (implied above), to restore the search order couldn't you 
do something similar to (assuming <mod_a> <mod_b> leave nothing on the 
stack):

: include-module  ( widn ... wid1 n "name" "file" -- )
    n>r module:
    include
    nr> set-order
;

get-order <mod_a> <mod_b>
include-module <mod_c> modc.fs

Not pretty but could perhaps be improved with a bit of syntax. If the 
file modc.fs leaves the stack unchanged you don't even need INCLUDE-MODULE

-- 
Gerry

On Oct 13, 2:06=A0pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> BruceMcF <agil...@netscape.net> wrote:

(and etc., odd ">"'s for Andrew, plain and even ">"'s for me.

> > So in addition to not wanting to have published library source using
> > PRIVATE and PUBLIC at all, can I take it that you would prefer any
> > library source that did so to use a system that compiled PUBLIC code
> > into the original compilation wordlist?

> Absolutely. =A0I think I said that already.

I had thought so, but in such a sprawling discussion, wanted to be
sure.

>>> When browsing, you look at Word X, which calls Word Y, which
>>> calls Word Z. You don't stop browsing at module boundaries or
>>> at private/public boundaries.

>> But your argument above seems stronger than that: not just that the
>> private/public boundary should not be a barrier to browsing, but that
>> the private/public should be invisible when browsing.

> > I can see the first, but I'm a bit skeptical about the latter.

> Well, invisible is ideal. =A0Anything that forces you to have to do
> extra work beyond thinking about your program itself, is, obviously,
> part of the downside.

But seamlessness and no extra work are two distinct design goals, with
the no extra work goal easier to reach than the broader seamlessness
design goal. For one thing, if the library module *has* special tools,
adding integration in the version of fsl-util for that specific
implementation that draws on those tools when useful might be a lot
less work to make no extra work than to make seamless.

> > For one thing, even when you want to browse beyond the boundary, you
> > are now entering a distinct system with its own mindset, a defined
> > glossary of public words, and hopefully a set of tests of the public
> > words compliance with their specified behavior. So you are browsing
> > a distinctive landscape with its own distinctive features, and that
> > context is useful information while browsing.

> > And of course, in a sufficiently large total namespace, not everyone
> > *will* want to browse beyond the public/private boundary of every
> > module ~ and if the boundary is invisible, they'll find themselves
> > browsing into territory they wouldn't have explored in that way if
> > they'd known.

> Perhaps so. =A0I rather appreciate the (non-Forth) tools I use all the
> time that quite happily let you e.g. step into "system" code. =A0It
> doesn't work for the Linux kernel, which really is the other side of a
> wall, but I'd consider that a bug.

And when, say, working on a tool to pull down the external streaming
video link together with series and episode information from the
episode streaming page of a bootleg leech video streaming site, I
can't imagine wanting to look at the system side of the network
access.

>> Now without any substantial published library source, we've got a
>> development process supported on a combination of the implementation
>> side and the development environment side to explore the application
>> side. With a substantial published library source, we have the same
>> combination to explore both the application side and the library
>> side.

>> You've got a strong preference for a seamless crossing of the
>> application side / library side boundary. I'm not sure I mind a seam,
>> if its not much trouble crossing it.

> That's not so bad.

But I normally browse the set of files that compile the app, so advise
on what problems may arise in browsing the dictionary is useful in
thinking it through.

With the right module search order control wordset, I think it would
be possible to automatically generate a master cross reference file,
and have a support tool that could be integrated into, eg, an extended
"locate" to avoid the blindside problem in dictionary exploration.

A search history log could be used to provide a "best guess".

Now, what do we need to make sure that reasonably well, requiring
little or no extra work on the part of the user? We need a "clean"
*public* namespace in the library, with no name clashes, so the module
can be inferred from the public word being examined.

And if we have a clean public namespace, the benefit of the library
encapsulating them in their own separate wordlist or wordlists goes
away ~ public name clashes becomes an issue of integrating the library
with the application and/or with other libraries in use, and either is
an application side integration issue.

But on other hand, that may just be me rationalizing the position I am
already leaning toward:
>> But with respect to the seamless crossing, I'd argue that its easier
>> to achieve if the application side / library side boundary is between
>> public and private than if the application side / library side
>> boundary is between the entire set of words defined by the library
>> module and the including application.

>>>> With the "named private" modular wordlist approach, if you are
>>>> developing the library, you only have to bring the private wordlist
>>>> into the search order and locate works just fine.

>>> Sheesh. How do you even know what the private wordlist is called?
>>> You're just browsing code. You don't want to have to search the files
>>> in which a word is defined to find out which private wordlist
>>> it's in.

>> If you are working *on* that module in the library, then you probably
>> want to know the name of the module independently of whether or not
>> you are browsing an instance of the compiled source.

> I doubt that very much. =A0Say I'm working on Module X, and Word Y
> doesn't seem to do the right thing. =A0The next thing I want to do is
> look at Word Y. =A0Do I want to know the name of the module in which Y
> is held before I even look at Y?

But at least, if you are working on that library, you'll know which
library, so nominating a "first check" master wordmap file for the
kind of extended "locate" that falls through into reference to the
master wordmap is something you could add to a project loadscript.

>> In any event, you're imagining that its not *been* made easy to do
>> so.

> I don't think I'm imagining anything. =A0I'm talking about the world as
> it is, right now, if you load words defined using such modules into a
> Forth system.

"such modules" do not yet exist, as the FSL is presently in its pooled-
private set-up, and making a transition would also include whatever
integration that the authors of the implementation-specific fsl-util
files would provide.

The problem you are describing is only an *ongoing* problem if its not
resolved by the author of the fsl-util file for the implementation
you're using.

>> My concern with that is whether it *could be* made easy to do so.
>> Since the file can easily be run through a filter that says what
>> modules are defined in it, and what private and public words are
>> defined in it, then so long as the semantics are adequate to account
>> for all words it defines *anywhere*, it *could be* made easy to do.

> I agree, it could be done, with a new set of tools.

My ideal would be a new set of tools that can be effectively
integrated into extensions of existing tools. Trying to do that
integration generically is impractical, so its a task for the
implementation library harness.

>>> OK, let's examine that more closely. We have a library that might be
>>> useful in some circumstances. We have a bunch of Forths and Forth
>>> programmers.

>>> As a user, I might want to use some of this library code. But why
>>> should I have a special toolset?

>> Why do we ever want tools designed for the job when we are working
>> on a specific task? Because working with tools designed for the task
>> make it easier to do the job.

> But as a *user* of the FSL you probably just want the FSL. =A0You don't
> necessarily want a bunch of new tools just because the designers of
> the FSL like to use modules.

But its also not the case that every *user* of the FSL will want to
browse its private words.

>>>?Is it your claim that FSL is only
>>> useful on Forths whose usual problem domain is one where a structured
>>> wordspace such a useful thing that special tools have been developed
>>> to cope with it? ?If so, I suspect FSL's potential user base is very
>>> small.

>> There's not much of an issue with FSL at present, since its PRIVATE
>> wordlist is a common pool, so you don't have to know which module a
>> private word is in in order to make sure that all private words are in
>> the search order ~ simply bring the single private wordlist into the
>> search order, and they're all visible.

> Good!

But as the FSL slowly grows, name clashes between private words become
a greater nuisance. Hence the original desire to have a distinct
private wordlist per module.

>>>> If you are doing a dictionary browse of a public word used
>>>> in one file but included from another, and then when looking
>>>> in it, you click on a word that is not in the search order,
>>>> but is located in the file that the word you are looking at
>>>> was included from ... popping up a second file viewer window
>>>> to look at where that word is located in the file would seem
>>>> to be a quite natural extension. It doesn't need to be all that
>>>> smart ~ hitting the search again control key or function key once
>>>> or twice will bring up the definition if the first hit is in the
>>>> descriptive documentation.

>>> Sounds cute but rather baroque, and it takes you away from
>>> the keyboard interpreter, which IME is always a distraction.

>> Yes, escaping the constraints and "looking at the source through a
>> mailbox slit" of the command line environment is the point of it,
>> which is why in Forth, screen based development environments have been
>> developed in addition to command line oriented development
>> environments for over forty years.

> I don't think this is anything to do with preferring "screen based
> development environments" to the command line, etc. =A0Sometimes one is
> appropriate, sometimes the other. =A0It's to do with *taking away* some
> capabilities of the command line.

If its *unintentionally* taking away some of the capabilities of the
command line, then it has a lot to do with preferred development
environment.

And if its an unintentional side-effect, rather than a deliberate
design goal, then it seems plausible that accommodation can be made to
avoid the problem.

>>>>> Indeed. It seems to me that the whole issue of how one would
>>>>> work on a system consisting of these modules has hitherto been
>>>>> ignored. Either the people working on such systems just put up
>>>>> with the Forth tools like LOCATE no longer working or they never
>>>>> learned to use them in the first place. "LOCATE doesn't work.
>>>>> Never mind, you can use grep." :-)

>>>> Or their Forth tools don't have any trouble with that.

>>> Maybe, but I doubt it. ?I think it's just been ignored.

>> For people who develop by shuttling back and forth between the command
>> line and the integrated source file editor, or shuttling back and
>> forth between their preferred general program editor and newly created
>> instance of Forth, they might not do much dictionary browsing ~

> Bingo! =A0That's *exactly* my point. =A0They don't do it, so they don't
> know that anyone else does it. =A0Unwittingly, they may destroy
> something useful because they don't understand. =A0That's why I'm
> spending all this time trying to explain.

My question is, how good is good enough?

(1) If the requirement is that the use of the module words, which is
an effective solution to a substantial standing problem in a portable
Forth source library, must portably and without any tools integration
make all of its PRIVATE words available to all existing host-side
development tools ...

.... the answer to that is to have a module semantics that eliminates
the namespace structure, placing the requirement for addressing any
namespace clashes that may arise on the user.

One issue where considering unstructured implementations shows up is
whether private wordlists from other modules are brought into the
search order by executing the wordlist directly, or by a parsing word
that operates on the name of the wordlist. In a flat wordlist, the
latter can be replaced by a dummy that parses and discards the module
name, since there's no module wordlist opening to do.

That eliminates problems in the event of a name clash between a module
name and another definition by the same name loaded in the interim,
once making sure that the module name is never executed directly by
source library code is adopted as a library-side integration
requirement.

(2) If the requirement is that the use of module words must *allow*
the integration of existing host-side tools to accommodate access to
the PRIVATE words ...

.... I think a master wordmap file offers one approach to doing that,
provided that each public word is uniquely associated with one module
in the library.

I assume that if there is an effective file based solution, there will
also exist effective solutions integrated into the dictionary, without
worrying so much about what shape they would take.

>> and then, mostly to squash unanticipated name-clash bugs, where the
>> PRIVATE/PUBLIC approach already tends to prevent those bugs from
>> occurring in the first place.

> Is most of your development time fixing unanticipated name-clash bugs?
> Really?

No. You can take the implication that my explicit use of "locate" on
the command line is mostly for that purpose, but nothing I've said
implies that I use locate a lot.

On Oct 13, 3:09=A0pm, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
wrote:
> get-order <mod_a> <mod_b>
> include-module <mod_c> modc.fs

> Not pretty but could perhaps be improved with a bit of syntax. If the
> file modc.fs leaves the stack unchanged you don't even
> need INCLUDE-MODULE

If the semantics of module: are to re-use an existing wordlist, and
only created a new wordlist if it doesn't exist, and does not use the
public module to store private wordlist info, this approach would
allow:

include-module FORTH-WORDLIST modc.fs

On Oct 13, 2:09=A0pm, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
wrote:
> On 13/10/2011 02:03, Krishna Myneni wrote:
>
> [...]
>
>
>
> > In my implementation, one should be able to do the following from the
> > "user" side:
>
> > MODULE: <name>
> > <mod_a> =A0<mod_b> =A0 =A0\ dependencies
> > include modc.fs
>
> We agree then.
>

I'm not advocating the above usage, just pointing out that the
existing implementation permits this so-called "user" mode, where the
module name and search order dependencies are set external to the
module code.  In the more rigid mode, both the module name and search
order dependencies are specified inside the module. I still think it
makes sense in a lot of cases for the module file to specify the
module's name, e.g. I see no reason the FSL file, horner.x, should not
contain the statement:

MODULE: FSL-Horner

as opposed to placing this statement in the external loading file. In
any case, the module.fs code, allows either style to be used.


>
>
> > In the above example, modc.fs will start with BEGIN-MODULE =A0and end
> > with END-MODULE , with PUBLIC: and PRIVATE: sections in between. In
> > this way, the module naming and setting up the dependencies is done on
> > the "user" side. However, the only drawback (from my perspective) is
> > that the search order is not restored to the state prior to the
> > dependencies line. One would have to do a PREVIOUS for each declared
> > dependency to return to the prior search order. Also, note that in my
> > implementation, MODULE: does not modify the search order or
> > compilation wordlist. It creates a named reference for the new module,
> > which goes into the compilation wordlist prior to the MODULE:
> > statement.
>
> If MODULE: doesn't change the search order (stated above) or record the
> search order (implied above), to restore the search order couldn't you
> do something similar to (assuming <mod_a> <mod_b> leave nothing on the
> stack):
>
> : include-module =A0( widn ... wid1 n "name" "file" -- )
> =A0 =A0 n>r module:
> =A0 =A0 include
> =A0 =A0 nr> set-order
> ;
>
> get-order <mod_a> <mod_b>
> include-module <mod_c> modc.fs
>

Yes, I think you should be able to do the above.

> Not pretty but could perhaps be improved with a bit of syntax. If the
> file modc.fs leaves the stack unchanged you don't even need INCLUDE-MODUL=
E
>

Not sure what you mean by the last statement. How can you exclude the
INCLUDE-MODULE statement and replace it with just
<mod_c> modc.fs? The name <mod_c> is undefined up to that point.

Krishna

On Oct 13, 3:00=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 13, 3:09=A0pm, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
> wrote:
>
> > get-order <mod_a> <mod_b>
> > include-module <mod_c> modc.fs
> > Not pretty but could perhaps be improved with a bit of syntax. If the
> > file modc.fs leaves the stack unchanged you don't even
> > need INCLUDE-MODULE
>
> If the semantics of module: are to re-use an existing wordlist, and
> only created a new wordlist if it doesn't exist, and does not use the
> public module to store private wordlist info, this approach would
> allow:
>
> include-module FORTH-WORDLIST modc.fs

Just to clarify. In my implementation, version 0.2 of module.fs,
MODULE: does absolutely nothing other than create a named reference to
the new module and reserve space for the public and private wordlists.
MODULE: does not create any new wordlists nor does it modify the
search order.

Krishna

On Oct 13, 6:26=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> I see no reason the FSL file, horner.x, should not
> contain the statement:

> MODULE: FSL-Horner
>
> as opposed to placing this statement in the external loading file. In
> any case, the module.fs code, allows either style to be used.

If the module file places PUBLIC words in "FSL-Horner", it should at
the very least be in a separate include file from the code, to make it
easier to directly compile the public words where the user desires.

I prefer that PUBLIC means "export to wherever directed by the
original compilation directory", as it seems to mean in Swiftforth
PACKAGES. , and as PUBLIC means in Neal Bridges' elegant little
modules.fs system when used as designed, called with the compilation
wordlist at the front of the search order.

On Oct 13, 6:28=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 13, 3:00=A0pm, BruceMcF <agil...@netscape.net> wrote:
>
> > On Oct 13, 3:09=A0pm, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
> > wrote:
>
> > > get-order <mod_a> <mod_b>
> > > include-module <mod_c> modc.fs
> > > Not pretty but could perhaps be improved with a bit of syntax. If the
> > > file modc.fs leaves the stack unchanged you don't even
> > > need INCLUDE-MODULE
>
> > If the semantics of module: are to re-use an existing wordlist, and
> > only created a new wordlist if it doesn't exist, and does not use the
> > public module to store private wordlist info, this approach would
> > allow:
>
> > include-module FORTH-WORDLIST modc.fs
>
> Just to clarify. In my implementation, version 0.2 of module.fs,
> MODULE: does absolutely nothing other than create a named reference to
> the new module and reserve space for the public and private wordlists.
> MODULE: does not create any new wordlists nor does it modify the
> search order.

The date of posting of version 0.2 would make it easier to find it,
its the Oct 5 OP that's easy to find. In that one, MODULE: is just
half of the named module function, completed by begin-module detecting
that a named module is halfway created and so doing the balance of the
function for a named module, while if begin-module does not detect
that a named module is halfway created, it do the complete process of
starting an anonymous module.

But it does create both wordlists.

In version 0.1 (?) I presume, it is when MODULE: is used in the base
library module that PUBLIC is converted from its normal meaning of
compiling in the original compilation directory into the alternate
meaning of compiling into a named vocabulary. If version 0.2 does the
same conversion, then MODULE: should not be in the base include file,
but in an optional MODULE: header file. It should be at least *easy*
to keep PUBLIC meaning what it has conventionally meant, by
sidestepping the defining of PUBLIC words into library-side defined
wordlists.

I expect there is a protocol that would allow the same BEGIN-
MODULE ... END-MODULE code to offer:
   inherited PUBLIC, anonymous PRIVATE
   inherited PUBLIC, pooled PRIVATE
   inherited PUBLIC, named individual PRIVATE
   named individual PUBLIC, named individual PRIVATE
   named grouped PUBLIC, named individual PRIVATE

By "named grouped PUBLIC", I mean collecting the public interfaces of
a group of modules into a single named wordlist, avoiding the "hundred
words split up into forty wordlists" problem.

Begin-Module ... End-Module side ... if public-wid holds 0, CURRENT is
the public-wid, initial-wid is 0, if public-wid holds a non-zero, that
remains the public-wid, initial-wid is CURRENT. ... if the private-wid
holds 0, a new private wordlist is created, if it holds a non-zero,
that remains the private-wid.

End-Module: if initial-wid is 0, public-wid become CURRENT, if initial-
wide is non-zero, it becomes CURRENT. If pool-private is 0, private-
wid is zeroed, if pool-private is TRUE, private-wid is retained.

reset-modules
  Sets pooled-private to 0, private-wid to 0, public-wid to 0.

PRIVATE: name
  creates vocabulary "name" in PRIVATE-MODULES if it does not exist,
sets private-wid to the wordlist in name.

POOLED-PRIVATE: name
  Does Private: and sets pooled-private to TRUE.

MODULE: name
  Sets public-wid to the wordlist in MODULES name.

MODULE-PAIR: name
  Performs both MODULE: and PRIVATE: on name.

* inherited PUBLIC, anonymous PRIVATE
   "reset-modules" and go.

* inherited PUBLIC, pooled PRIVATE
   "POOLED-PRIVATE: name" and go.

* inherited PUBLIC, named individual PRIVATE
   "PRIVATE: name" and go.

* named individual PUBLIC, named individual PRIVATE
   "MODULE-PAIR: name" and go

* named grouped PUBLIC, named individual PRIVATE
   "MODULE: groupname PRIVATE: modulename" and go.

An advantage is that, because both the public and private wordlists
are host vocabularies, they show up in ORDER if the host shows
vocabulary names in ORDER. A pair of words are defined:
   <voc vocabularyname voc>
.... that have the net effect of extracting the vocabulary name to the
stack without affecting the search order. If the host vocabulary is
F83 style, <voc is ALSO and voc> is order>, while if the host
vocabulary is self-pushing, <voc is a no-op, while voc> is order>.

On 13/10/2011 23:26, Krishna Myneni wrote:
> On Oct 13, 2:09 pm, Gerry Jackson<ge...@jackson9000.fsnet.co.uk>
> wrote:
>> On 13/10/2011 02:03, Krishna Myneni wrote:
>>
>> [...]
>>
>>
>>
>>> In my implementation, one should be able to do the following from the
>>> "user" side:
>>
>>> MODULE:<name>
>>> <mod_a>    <mod_b>      \ dependencies
>>> include modc.fs
>>
>> We agree then.
>>
>
> I'm not advocating the above usage, just pointing out that the
> existing implementation permits this so-called "user" mode, where the
> module name and search order dependencies are set external to the
> module code.  In the more rigid mode, both the module name and search
> order dependencies are specified inside the module. I still think it
> makes sense in a lot of cases for the module file to specify the
> module's name, e.g. I see no reason the FSL file, horner.x, should not
> contain the statement:
>
> MODULE: FSL-Horner
>
> as opposed to placing this statement in the external loading file. In
> any case, the module.fs code, allows either style to be used.
>

It's good that it is flexible enough to allow either style. If a module 
names itself your MODULE: enables a user to give the module another name 
if he so wishes e.g. to avoid a module name clash, to include a version 
number or simply because he doesn't like the given name.

>>> In the above example, modc.fs will start with BEGIN-MODULE  and end
>>> with END-MODULE , with PUBLIC: and PRIVATE: sections in between. In
>>> this way, the module naming and setting up the dependencies is done on
>>> the "user" side. However, the only drawback (from my perspective) is
>>> that the search order is not restored to the state prior to the
>>> dependencies line. One would have to do a PREVIOUS for each declared
>>> dependency to return to the prior search order. Also, note that in my
>>> implementation, MODULE: does not modify the search order or
>>> compilation wordlist. It creates a named reference for the new module,
>>> which goes into the compilation wordlist prior to the MODULE:
>>> statement.
>>
>> If MODULE: doesn't change the search order (stated above) or record the
>> search order (implied above), to restore the search order couldn't you
>> do something similar to (assuming<mod_a>  <mod_b>  leave nothing on the
>> stack):
>>
>> : include-module  ( widn ... wid1 n "name" "file" -- )
>>      n>r module:
>>      include
>>      nr>  set-order
>> ;
>>
>> get-order <mod_a>  <mod_b>
>> include-module <mod_c>  modc.fs
>>
>
> Yes, I think you should be able to do the above.
>
>> Not pretty but could perhaps be improved with a bit of syntax. If the
>> file modc.fs leaves the stack unchanged you don't even need INCLUDE-MODULE
>>
>
> Not sure what you mean by the last statement. How can you exclude the
> INCLUDE-MODULE statement and replace it with just
> <mod_c>  modc.fs? The name <mod_c>  is undefined up to that point.

I meant that you could write:

get-order <mod_a> <mod_b>
module: <mod_c>
include modc.fs
set-order

-- 
Gerry

BruceMcF <agila61@netscape.net> wrote:
> 
> But as the FSL slowly grows, name clashes between private words become
> a greater nuisance. Hence the original desire to have a distinct
> private wordlist per module.

Let's pause to examine that.  What nuisance is caused by name clashes
between private words?  I can't think of anything.
 
>> Bingo! ?That's *exactly* my point. ?They don't do it, so they don't
>> know that anyone else does it. Unwittingly, they may destroy
>> something useful because they don't understand. That's why I'm
>> spending all this time trying to explain.
> 
> My question is, how good is good enough?
> 
> (1) If the requirement is that the use of the module words, which is
> an effective solution to a substantial standing problem

That's a rather grand claim.

>>> and then, mostly to squash unanticipated name-clash bugs, where the
>>> PRIVATE/PUBLIC approach already tends to prevent those bugs from
>>> occurring in the first place.
> 
>> Is most of your development time fixing unanticipated name-clash bugs?
>> Really?
> 
> No. You can take the implication that my explicit use of "locate" on
> the command line is mostly for that purpose, but nothing I've said
> implies that I use locate a lot.

Oh, I see.  I think this exchange has revealed more than I expected.
People really do program in very different ways.  When browsing I
don't much care where words are defined, as long as LOCATE gets me
their source code.  Having to search a text file for a definition
seems really stone-age!  I'm vaguely in favour of grouping words
together, but I suppose it doesn't affect me as much as others.

Andrew.

On Oct 13, 5:46=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 13, 6:26=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > I see no reason the FSL file, horner.x, should not
> > contain the statement:
> > MODULE: FSL-Horner
>
> > as opposed to placing this statement in the external loading file. In
> > any case, the module.fs code, allows either style to be used.
>
> If the module file places PUBLIC words in "FSL-Horner", it should at
> the very least be in a separate include file from the code, to make it
> easier to directly compile the public words where the user desires.
>

In the module file, use of MODULE: creates a named reference to the
PUBLIC words in the *current compilation wordlist*. This already gives
the user sufficient control for accessing the words. I don't
understand why this one extra level of indirection is so troubling,
i.e. why you are insistent that the named module's PUBLIC words *must*
go into the current compilation wordlist (such a result can be
achieved by not naming the module). Similarly, the PRIVATE words of a
module can be made visible through the module's name as well. This
involves one extra level of indirection over making the PUBLIC words
visible, but that seems very reasonable to me.

> I prefer that PUBLIC means "export to wherever directed by the
> original compilation directory", as it seems to mean in Swiftforth
> PACKAGES. , and as PUBLIC means in Neal Bridges' elegant little
> modules.fs system when used as designed, called with the compilation
> wordlist at the front of the search order.


Krishna

On Oct 14, 1:50=A0am, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
wrote:
> On 13/10/2011 23:26, Krishna Myneni wrote:
>
>
>
> > On Oct 13, 2:09 pm, Gerry Jackson<ge...@jackson9000.fsnet.co.uk>
> > wrote:
> >> On 13/10/2011 02:03, Krishna Myneni wrote:
>
> >> [...]
>
> >>> In my implementation, one should be able to do the following from the
> >>> "user" side:
>
> >>> MODULE:<name>
> >>> <mod_a> =A0 =A0<mod_b> =A0 =A0 =A0\ dependencies
> >>> include modc.fs
>
> >> We agree then.
>
> > I'm not advocating the above usage, just pointing out that the
> > existing implementation permits this so-called "user" mode, where the
> > module name and search order dependencies are set external to the
> > module code. =A0In the more rigid mode, both the module name and search
> > order dependencies are specified inside the module. I still think it
> > makes sense in a lot of cases for the module file to specify the
> > module's name, e.g. I see no reason the FSL file, horner.x, should not
> > contain the statement:
>
> > MODULE: FSL-Horner
>
> > as opposed to placing this statement in the external loading file. In
> > any case, the module.fs code, allows either style to be used.
>
> It's good that it is flexible enough to allow either style. If a module
> names itself your MODULE: enables a user to give the module another name
> if he so wishes e.g. to avoid a module name clash, to include a version
> number or simply because he doesn't like the given name.
>

With a minor change to the definition of MODULE: , I see a way to
allow the user to override the name of a module, given inside the
module file. For example, if horner.x contained the statement,

MODULE: FSL-Horner

the named reference FSL-Horner will be created in the current
compilation wordlist when horner.x is included. However, if the user
wants a different module name, to avoid a name clash, it should be
possible to do:

MODULE: FSL-Horner-1.5c
include horner.x

even though horner.x already names the module. The required change to
the definition of MODULE: to make the external renaming possible (when
an intrinsic name is already given) is:

\ Create the name for a named module and reserve space
\ to store public and private wordlists
: MODULE:  ( "name" -- )
  named-module @ 0=3D IF
    CREATE  here named-module !  0 , 0 ,
    DOES>  ( o: -- wid )  @ >order  \ add public wordlist to search
order
  THEN
;

I haven't tested this, however, I think it will work.


>
>
> >>> In the above example, modc.fs will start with BEGIN-MODULE =A0and end
> >>> with END-MODULE , with PUBLIC: and PRIVATE: sections in between. In
> >>> this way, the module naming and setting up the dependencies is done o=
n
> >>> the "user" side. However, the only drawback (from my perspective) is
> >>> that the search order is not restored to the state prior to the
> >>> dependencies line. One would have to do a PREVIOUS for each declared
> >>> dependency to return to the prior search order. Also, note that in my
> >>> implementation, MODULE: does not modify the search order or
> >>> compilation wordlist. It creates a named reference for the new module=
,
> >>> which goes into the compilation wordlist prior to the MODULE:
> >>> statement.
>
> >> If MODULE: doesn't change the search order (stated above) or record th=
e
> >> search order (implied above), to restore the search order couldn't you
> >> do something similar to (assuming<mod_a> =A0<mod_b> =A0leave nothing o=
n the
> >> stack):
>
> >> : include-module =A0( widn ... wid1 n "name" "file" -- )
> >> =A0 =A0 =A0n>r module:
> >> =A0 =A0 =A0include
> >> =A0 =A0 =A0nr> =A0set-order
> >> ;
>
> >> get-order <mod_a> =A0<mod_b>
> >> include-module <mod_c> =A0modc.fs
>
> > Yes, I think you should be able to do the above.
>
> >> Not pretty but could perhaps be improved with a bit of syntax. If the
> >> file modc.fs leaves the stack unchanged you don't even need INCLUDE-MO=
DULE
>
> > Not sure what you mean by the last statement. How can you exclude the
> > INCLUDE-MODULE statement and replace it with just
> > <mod_c> =A0modc.fs? The name <mod_c> =A0is undefined up to that point.
>
> I meant that you could write:
>
> get-order <mod_a> <mod_b>
> module: <mod_c>
> include modc.fs
> set-order
>
> --
> Gerry

Yes. I see now. That should work, and it is pretty close to what one
would do if using VOCABULARYs, except it's cleaner.

Krishna

On Oct 14, 6:23=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 14, 1:50=A0am, Gerry Jackson <ge...@jackson9000.fsnet.co.uk>
> wrote:
>
....
>
> With a minor change to the definition of MODULE: , I see a way to
> allow the user to override the name of a module, given inside the
> module file. For example, if horner.x contained the statement,
>
> MODULE: FSL-Horner
>
> the named reference FSL-Horner will be created in the current
> compilation wordlist when horner.x is included. However, if the user
> wants a different module name, to avoid a name clash, it should be
> possible to do:
>
> MODULE: FSL-Horner-1.5c
> include horner.x
>
> even though horner.x already names the module. The required change to
> the definition of MODULE: to make the external renaming possible (when
> an intrinsic name is already given) is:..

There's a problem with the external renaming, however. Other modules
which declare a dependency on FSL-Horner will not be able to find the
dependency, if the module name is changed externally. If the module
name is simply given a synonym, e.g.

include horner.x
SYNONYM FSL-Horner FSL-Horner-1.5c

then no subsequent dependencies are broken, and either name can be
used to reference the module's public interface. However, the synonym
method would not avoid a name clash. I doubt it will be a problem for
something like the FSL, but I can see where such a problem might arise
for something more generic such as a library of graphics related
modules. For generic libraries, the external naming and dependency
declaration seems preferable, e.g. Gerry's example:

get-order <mod_a> <mod_b>
module: <mod_c>
include modc.fs
set-order


Krishna

On Oct 14, 7:02=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:

> In the module file, use of MODULE: creates a named reference to the
> PUBLIC words in the *current compilation wordlist*.

Yes. But its the PUBLIC words that the application wants from the
library ~ the named "public" modules are just extra clutter and a
barrier placed between the application and what it wants from the
library.

Now, its one thing if the library is *offering* a namespace structure
to the application, that it can use or not as it wishes. But it should
not be *imposing* structure on the application side namespace.

Below, you ask what is wrong with the library imposing structure on
the application side namespace, but you haven't yet said what is
*right* about it? I would think in proposing such a dramatic break in
the meaning of a word as commonplace as PUBLIC, the burden of proof
should be on the person proposing the dramatic break.

In the commonplace meaning of PUBLIC as implemented in a variety of
ways, the library putting them into modules means they aren't actually
public words, they are words in a named private wordlist.

As far as specific reasons why not ...

Andrew Haley has given one reason at length ~ when using the existing
implementation tools to browse the code for a given word that has been
stored in the dictionary when that word was defined, there is a long
standing presumption that the words you want to browse are visible.
The more you've placed the words you wish to browse into a multitude
of individual named wordlists, the more you impose a requirement on
the person using those tools to learn the details of how the library
is put together in order to explore it.

The second-best to him seems to be a single private wordlist per
library, so there is only one thing to know to make almost everything
visible.

Also, Forth Inc. have found that caller-controlled PUBLIC and named
private suffices for all of their modular development needs.

There is also the issue of how it would work if it *is* adopted more
broadly. After all, you have long since passed the objectives required
for a one-off module programming system for a single library: you are
now working toward a general modular programming system, with the FSL
as the vehicle for seeding it into a range of existing applications,
and that presuming it meets the needs of another library, that would
make it the de facto choice for anyone else requiring a modular
programming system for an open source library, since they can first
test if the FSL system is already included, and use it if it is.

Consider an application that is put together with display code in one
or more display modules, user interface code in several user interface
modules, math support from a math library. Suppose for simplicity
that, wonder of wonders, they all use the same library module system.
And suppose that the application itself is built in six application
side modules.

The portable assumption is that there are eight levels available in
the search order, and one or two might be used by the original set-up.
In any module programming system where the PUBLIC words are not going
straight to FORTH, there are at least two more used for the PUBLIC
compilation wordlist and private compilation wordlist. So there are
four namespaces generically available to be brought into search order
context. Of course, in many implementations the search order may be
deeper than that, but when putting together portable source, whatever
can be done within the guaranteed minimums is one less implementation
dependency.

With modules with their PUBLIC words pooled directly into larger
classes, up to four general classes of facilities is ample. With each
distinct module have its own PUBLIC wordlist, its a royal pain.

Indeed, the module itself creating the wordlists for the public words
intrinsically creates interference with a different preferred
structure. This is not an issue for the common scenario where there
*is* not structure on the application side ~ but if there is, there's
interference.

An example: you've got a collection of user interface modules that
give the same public words, but in private work on iOS, Linux-Android,
Windows Phone and Samsung's bada. With application side named-wordlist
control, the implementation can put all of the public words for each
system in a common wordlist, and then pick compilation target by just
bringing the correct system's wordlist into context on the search
order. With emulators for all four in your development system, you've
got a much more convenient cross platform app factory for, eg,
smartphone device control apps than if each individual module in each
individual task class had private words named by the module instead of
actual PUBLIC words.

> This already gives
> the user sufficient control for accessing the words.

But it does not give the user sufficient control in the structure of
the application namespace. Either (1) leave the structure of the
application side namespace entirely in the hands of the caller, or (2)
offer the tools in the modular programming toolkit so that they easily
impose an alternate structure if they wish.

(1) is obviously simpler ~ but (2) is also workable, if the library
creating the application side namespace structure for its public words
happens in a different file that loads the base modules.

> I don't
> understand why this one extra level of indirection is so troubling,
> i.e. why you are insistent that the named module's PUBLIC words *must*
> go into the current compilation wordlist (such a result can be
> achieved by not naming the module).

Except you only get the freedom from the libraries naming structure at
the cost of burying all capabilities represented by an anonymous, one-
shot private wordlist.

So if you *want*, precisely, the current FSL structure, you can't have
it. Or if you *want*, precisely, the structure resulting from
SwiftForth's PACKAGE system, you can't have it ~ except of course if
you replace the FSL system with an alternative.

> Similarly, the PRIVATE words of a
> module can be made visible through the module's name as well.

But only at the cost of accepting the modules structuring of the
application's namespace. The application should be in control of the
structure of its namespace.

> This
> involves one extra level of indirection over making the PUBLIC words
> visible, but that seems very reasonable to me.

But its not a modular programming system that I could adopt
unmodified, because it does not permit me to give the control to the
library user that I would wish to give. Especially as I would be far
more likely to be adapting existing open source and public domain
where the primary value of the library would be convenience of use ~
FSL users might be willing to grudgingly put up with the inconvenience
of the library interfering with their application namespace structure,
but I doubt anything I could cobble together would fall in the same
boat.

> > I prefer that PUBLIC means "export to wherever directed by the
> > original compilation directory", as it seems to mean in Swiftforth
> > PACKAGES. , and as PUBLIC means in Neal Bridges' elegant little
> > modules.fs system when used as designed, called with the compilation
> > wordlist at the front of the search order.

Indeed, if you want to make a system targeted to switching between
private and hidden words, why not just call what you are calling
PRIVATE, "HIDDEN", and call what you are calling PUBLIC, "PRIVATE". If
the system is not set up to actually provide PUBLIC words, it should
not be labeling sections *as* PUBLIC.

On Oct 14, 5:19=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> BruceMcF <agil...@netscape.net> wrote:

>> But as the FSL slowly grows, name clashes between private words become
>> a greater nuisance. Hence the original desire to have a distinct
>> private wordlist per module.

> Let's pause to examine that. =A0What nuisance is caused by name clashes
> between private words? =A0I can't think of anything.

The specific example given was the same floating point variable name
is used in the PRIVATE section of two different modules. One
contribution has a typo in the variable name. When the library
maintainer tests the module with the typo, it passes ~ and passes even
if tested against multiple implementations, because the problem is not
an implementation dependency but the existence of the variable in the
other module. But then when the library user *either* tries to use
both modules together, things break because both assume they have
exclusive use of the private variable, or use the module with the typo
without including the library with the correct name, the module fails
to load at all.

The more mistakes automated testing without a lot of initial set-up
can catch, the better, and the simplest set-up is to load the standing
library, then the modified module, then run the module text script on
the modified module.

If public library maintenance is normally an unpaid spare time
volunteer activity, taking a set of library integration problems off
the table is of some actual benefit.

> >> Bingo! ?That's *exactly* my point. ?They don't do it, so they don't
> >> know that anyone else does it. Unwittingly, they may destroy
> >> something useful because they don't understand. That's why I'm
> >> spending all this time trying to explain.
>
> > My question is, how good is good enough?

> > (1) If the requirement is that the use of the module words, which is
> > an effective solution to a substantial standing problem

> That's a rather grand claim.

Just because you don't work in problem domains where the problem
arises does not imply that the problem does not arise. Of course, I'd
never argue they are universal problems ~ but if they were not
standing problems, solutions to the problems would not have been
repeatedly re-invented.

> >>> and then, mostly to squash unanticipated name-clash bugs, where the
> >>> PRIVATE/PUBLIC approach already tends to prevent those bugs from
> >>> occurring in the first place.
>
>>> Is most of your development time fixing unanticipated name-clash
>>> bugs? Really?

> > No. You can take the implication that my explicit use of "locate" on
> > the command line is mostly for that purpose, but nothing I've said
> > implies that I use locate a lot.

> Oh, I see. =A0I think this exchange has revealed more than I expected.
> People really do program in very different ways. =A0When browsing I
> don't much care where words are defined, as long as LOCATE gets me
> their source code. =A0Having to search a text file for a definition
> seems really stone-age!

It may be point and drool, but clicking on a word and hitting a
function key is quicker than typing "locate" as well as the word name.
If its more than a click or two and a keypress or two, its not the
searching that is stone age so much as the source file editor.

>=A0I'm vaguely in favour of grouping words
> together, but I suppose it doesn't affect me as much as others.

On the other thread under this heading, I've argued there are two
general approaches to allowing the users of a library to retain the
control over the structure of their application namespace while
including published library code that includes a modular compilation
system.

(1) The general Swiftforth approach in "PACKAGE", where the entire
control of where public words are compiled belongs to the caller ~ the
library simply does not give a structure to the public namespace.

(2) A small, flexible toolkit that allows the library to to put the
default public word structure in one file that includes the underlying
module definitions, so that a different loadscript, or no loadscript
at all, can give a different structure.

I'm thinking that (2) addresses your concern most directly for an
application that loads the modules themselves, since it would allow
just setting up a single namespace for all private words using that
modular system. I could sketch more seamless and complicated
integrations, but in any event, in the same loadscript define a word
("get-private") to bring that namespace into the search order and
bob's your uncle.

BruceMcF <agila61@netscape.net> wrote:
> On Oct 14, 5:19?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>> BruceMcF <agil...@netscape.net> wrote:
> 
>>> But as the FSL slowly grows, name clashes between private words become
>>> a greater nuisance. Hence the original desire to have a distinct
>>> private wordlist per module.
> 
>> Let's pause to examine that. ?What nuisance is caused by name clashes
>> between private words? ?I can't think of anything.
> 
> The specific example given was the same floating point variable name
> is used in the PRIVATE section of two different modules. One
> contribution has a typo in the variable name. When the library
> maintainer tests the module with the typo, it passes ~ and passes even
> if tested against multiple implementations, because the problem is not
> an implementation dependency but the existence of the variable in the
> other module. But then when the library user *either* tries to use
> both modules together, things break because both assume they have
> exclusive use of the private variable, or use the module with the typo
> without including the library with the correct name, the module fails
> to load at all.

OK, so it requires a typo, not just a name clash.  Fair enough.

> If public library maintenance is normally an unpaid spare time
> volunteer activity, taking a set of library integration problems off
> the table is of some actual benefit.
> 
>> >> Bingo! ?That's *exactly* my point. ?They don't do it, so they don't
>> >> know that anyone else does it. Unwittingly, they may destroy
>> >> something useful because they don't understand. That's why I'm
>> >> spending all this time trying to explain.
>>
>> > My question is, how good is good enough?
> 
>> > (1) If the requirement is that the use of the module words, which is
>> > an effective solution to a substantial standing problem
> 
>> That's a rather grand claim.
> 
> Just because you don't work in problem domains where the problem
> arises does not imply that the problem does not arise.

Or that it does; it's your claim.

quod gratis asseritur, gratis negatur: what is asserted without reason
may be denied without reason.

>> > No. You can take the implication that my explicit use of "locate" on
>> > the command line is mostly for that purpose, but nothing I've said
>> > implies that I use locate a lot.
> 
>> Oh, I see. I think this exchange has revealed more than I expected.
>> People really do program in very different ways. When browsing I
>> don't much care where words are defined, as long as LOCATE gets me
>> their source code. Having to search a text file for a definition
>> seems really stone-age!
> 
> It may be point and drool, but clicking on a word and hitting a
> function key is quicker than typing "locate" as well as the word name.

It's not so much point and drool as old school.  Searching in the
editor is what we did before LOCATE .

> If its more than a click or two and a keypress or two, its not the
> searching that is stone age so much as the source file editor.

OK, but without knowing the dictionary search context (or perhaps
decompiling the definition) how does the editor know where in the
source code a word is defined?  There may be all manner of conditional
compliation, words generated on the fly, and so on.  That's the thing
about LOCATE: it's always right.

>> I'm vaguely in favour of grouping words together, but I suppose it
>> doesn't affect me as much as others.
> 
> On the other thread under this heading, I've argued there are two
> general approaches to allowing the users of a library to retain the
> control over the structure of their application namespace while
> including published library code that includes a modular compilation
> system.
> 
> (1) The general Swiftforth approach in "PACKAGE", where the entire
> control of where public words are compiled belongs to the caller ~ the
> library simply does not give a structure to the public namespace.
> 
> (2) A small, flexible toolkit that allows the library to to put the
> default public word structure in one file that includes the underlying
> module definitions, so that a different loadscript, or no loadscript
> at all, can give a different structure.
> 
> I'm thinking that (2) addresses your concern most directly for an
> application that loads the modules themselves, since it would allow
> just setting up a single namespace for all private words using that
> modular system.

Right.

> I could sketch more seamless and complicated integrations, but in
> any event, in the same loadscript define a word ("get-private") to
> bring that namespace into the search order and bob's your uncle.

Sounds good.

Andrew.

On Oct 14, 7:39=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> There's a problem with the external renaming, however. Other modules
> which declare a dependency on FSL-Horner will not be able to find the
> dependency, if the module name is changed externally. If the module
> name is simply given a synonym, e.g.

> include horner.x
> SYNONYM FSL-Horner FSL-Horner-1.5c

> then no subsequent dependencies are broken, and either name can be
> used to reference the module's public interface.

One approach to that is to take the dependency management out of the
module file and into the loadscript.

In the context of MODULE: ... BEGIN-MODULE ... END-MODULE, END-MODULE
would set the prior-depth to 0 after using it, MODULE: would set prior-
depth, and BEGIN-MODULE would only set module-depth if it is presently
0.

For standard declaration of dependencies in the underlying module,
consider putting a default MODULE: and dependency section in a
conditionally compiled block.

: [IFDEFINED] ( "name" -- flag )
   \ *G Execute named constant if it exists, otherwise return 0
   BL WORD FIND IF EXECUTE ELSE 2DROP FALSE THEN ;

....

[IFDEFINED] FSL-default [IF]
   MODULE: foo
   <voc bar1
   <voc bar2
   <voc bar3

[THEN] BEGIN-MODULE
 ...
END-MODULE

jacko wrote:
> Where do the hidden words go? From the point of view of the adapter, surely they must remain visible
> somewhere.

Hidden words don't go anywhere.

Word-hiding works on the premise that certain words are used only
during creation of a library or app, after which they may be rendered
invisible.

There are various ways one can make a word invisible e.g. unlinking
its name or putting it in a special wordlist created for the purpose.
Either way, if you think you may need the word later then don't hide it.

Indeed, closed modules and hidden words could be seen as antithetical
to Forth since factoring produces many potentially useful and re-usable
words.

In article <9391aac2-bd46-4185-885b-c891b9ced5c5@hv4g2000vbb.googlegroups.com>,
BruceMcF  <agila61@netscape.net> wrote:
>
>That eliminates problems in the event of a name clash between a module
>name and another definition by the same name loaded in the interim,
>once making sure that the module name is never executed directly by
>source library code is adopted as a library-side integration
>requirement.

This amounts to not using private names in MACRO's where those
private names are used in other modules too.
A big no no (as Anton Ertl will certainly agree to.)
This, of course, happens minly if the library writer delays
use till run time, in this fasion.

Groetjes Albert

--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

In article <j7bd27$qu$1@news-01.bur.connect.com.au>,
Ed <nospam@invalid.com> wrote:
>jacko wrote:
>> Where do the hidden words go? From the point of view of the adapter, surely they must remain visible
>> somewhere.
>
>Hidden words don't go anywhere.
>
>Word-hiding works on the premise that certain words are used only
>during creation of a library or app, after which they may be rendered
>invisible.
>
>There are various ways one can make a word invisible e.g. unlinking
>its name or putting it in a special wordlist created for the purpose.
>Either way, if you think you may need the word later then don't hide it.
>
>Indeed, closed modules and hidden words could be seen as antithetical
>to Forth since factoring produces many potentially useful and re-usable
>words.

As the use of libraries is mainly the situation where you don't
want to see the internals of modules, this can be construed as
"library use is antithetical to Forth".

Maybe it is. If I look at the "library" in my Forth (.lab) it
is tree like and it is conceived as a whole. If you want a
branch, you pull in all branches that are used, unless it
is already there. Whether you have a facility, is checked
by ... looking up the word itself. If it is not there, it
must be loaded.

It is almost like

117 LOAD
and in screen 117
   23 LOAD 111 LOAD
   ...
and in screen 23
  18 4 THRU
  ...

but without the screen numbers (and no double loading of
the same word.)

Groetjes Albert

--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

On Oct 15, 3:36=A0am, "Ed" <nos...@invalid.com> wrote:
> jacko wrote:
> > Where do the hidden words go? From the point of view of the adapter, su=
rely they must remain visible
> > somewhere.
>
> Hidden words don't go anywhere.
>
> Word-hiding works on the premise that certain words are used only
> during creation of a library or app, after which they may be rendered
> invisible.
>
> There are various ways one can make a word invisible e.g. unlinking
> its name or putting it in a special wordlist created for the purpose.
> Either way, if you think you may need the word later then don't hide it.
>

Sure, and the modular programming code we have published is one way of
doing that. If that seems excessively complex to you, by all means,
use a simpler approach.


> Indeed, closed modules and hidden words could be seen as antithetical
> to Forth since factoring produces many potentially useful and re-usable
> words.

In the real world, programmers sometimes have to interface to things
outside of the Forth environment. Let's take a concrete example:

ftp://ccreweb.org/software/kforth/examples/gpib.4th

The above is an interface to an external device driver, written in C,
for a General Purpose Interface Bus. It is impractical to consider
coding the actual device driver in Forth (using the Forth's assembler,
for example), and then one would have to operate with root privileges
-- of course you may want to write your own Forth operating system and
use that. By all means, do so.

But, going back to the above example, it is clear that the driver-
dependent constants are used only by the interface words. There's no
need for constants like these to be visible to the user and clutter
the search order, *once the interface is debugged*. The concept of a
module is that it is a reusable block of code with a public interface
that has been tested.

Krishna

On Oct 14, 12:41=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 14, 7:02=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > In the module file, use of MODULE: creates a named reference to the
> > PUBLIC words in the *current compilation wordlist*.
>
> Yes. But its the PUBLIC words that the application wants from the
> library ~ the named "public" modules are just extra clutter and a
> barrier placed between the application and what it wants from the
> library.
>
> Now, its one thing if the library is *offering* a namespace structure
> to the application, that it can use or not as it wishes. But it should
> not be *imposing* structure on the application side namespace.
>
> Below, you ask what is wrong with the library imposing structure on
> the application side namespace, but you haven't yet said what is
> *right* about it? I would think in proposing such a dramatic break in
> the meaning of a word as commonplace as PUBLIC, the burden of proof
> should be on the person proposing the dramatic break.
>
> In the commonplace meaning of PUBLIC as implemented in a variety of
> ways, the library putting them into modules means they aren't actually
> public words, they are words in a named private wordlist.
>
> As far as specific reasons why not ...
>
> Andrew Haley has given one reason at length ~ when using the existing
> implementation tools to browse the code for a given word that has been
> stored in the dictionary when that word was defined, there is a long
> standing presumption that the words you want to browse are visible.
> The more you've placed the words you wish to browse into a multitude
> of individual named wordlists, the more you impose a requirement on
> the person using those tools to learn the details of how the library
> is put together in order to explore it.
>
> The second-best to him seems to be a single private wordlist per
> library, so there is only one thing to know to make almost everything
> visible.
>
> Also, Forth Inc. have found that caller-controlled PUBLIC and named
> private suffices for all of their modular development needs.
>
> There is also the issue of how it would work if it *is* adopted more
> broadly. After all, you have long since passed the objectives required
> for a one-off module programming system for a single library: you are
> now working toward a general modular programming system, with the FSL
> as the vehicle for seeding it into a range of existing applications,
> and that presuming it meets the needs of another library, that would
> make it the de facto choice for anyone else requiring a modular
> programming system for an open source library, since they can first
> test if the FSL system is already included, and use it if it is.
>
> Consider an application that is put together with display code in one
> or more display modules, user interface code in several user interface
> modules, math support from a math library. Suppose for simplicity
> that, wonder of wonders, they all use the same library module system.
> And suppose that the application itself is built in six application
> side modules.
>
> The portable assumption is that there are eight levels available in
> the search order, and one or two might be used by the original set-up.
> In any module programming system where the PUBLIC words are not going
> straight to FORTH, there are at least two more used for the PUBLIC
> compilation wordlist and private compilation wordlist. So there are
> four namespaces generically available to be brought into search order
> context. Of course, in many implementations the search order may be
> deeper than that, but when putting together portable source, whatever
> can be done within the guaranteed minimums is one less implementation
> dependency.
>
> With modules with their PUBLIC words pooled directly into larger
> classes, up to four general classes of facilities is ample. With each
> distinct module have its own PUBLIC wordlist, its a royal pain.
>
> Indeed, the module itself creating the wordlists for the public words
> intrinsically creates interference with a different preferred
> structure. This is not an issue for the common scenario where there
> *is* not structure on the application side ~ but if there is, there's
> interference.
>
> An example: you've got a collection of user interface modules that
> give the same public words, but in private work on iOS, Linux-Android,
> Windows Phone and Samsung's bada. With application side named-wordlist
> control, the implementation can put all of the public words for each
> system in a common wordlist, and then pick compilation target by just
> bringing the correct system's wordlist into context on the search
> order. With emulators for all four in your development system, you've
> got a much more convenient cross platform app factory for, eg,
> smartphone device control apps than if each individual module in each
> individual task class had private words named by the module instead of
> actual PUBLIC words.
>
> > This already gives
> > the user sufficient control for accessing the words.
>
> But it does not give the user sufficient control in the structure of
> the application namespace. Either (1) leave the structure of the
> application side namespace entirely in the hands of the caller, or (2)
> offer the tools in the modular programming toolkit so that they easily
> impose an alternate structure if they wish.
>
> (1) is obviously simpler ~ but (2) is also workable, if the library
> creating the application side namespace structure for its public words
> happens in a different file that loads the base modules.
>
> > I don't
> > understand why this one extra level of indirection is so troubling,
> > i.e. why you are insistent that the named module's PUBLIC words *must*
> > go into the current compilation wordlist (such a result can be
> > achieved by not naming the module).
>
> Except you only get the freedom from the libraries naming structure at
> the cost of burying all capabilities represented by an anonymous, one-
> shot private wordlist.
>
> So if you *want*, precisely, the current FSL structure, you can't have
> it. Or if you *want*, precisely, the structure resulting from
> SwiftForth's PACKAGE system, you can't have it ~ except of course if
> you replace the FSL system with an alternative.
>
> > Similarly, the PRIVATE words of a
> > module can be made visible through the module's name as well.
>
> But only at the cost of accepting the modules structuring of the
> application's namespace. The application should be in control of the
> structure of its namespace.
>
> > This
> > involves one extra level of indirection over making the PUBLIC words
> > visible, but that seems very reasonable to me.
>
> But its not a modular programming system that I could adopt
> unmodified, because it does not permit me to give the control to the
> library user that I would wish to give. Especially as I would be far
> more likely to be adapting existing open source and public domain
> where the primary value of the library would be convenience of use ~
> FSL users might be willing to grudgingly put up with the inconvenience
> of the library interfering with their application namespace structure,
> but I doubt anything I could cobble together would fall in the same
> boat.
>
> > > I prefer that PUBLIC means "export to wherever directed by the
> > > original compilation directory", as it seems to mean in Swiftforth
> > > PACKAGES. , and as PUBLIC means in Neal Bridges' elegant little
> > > modules.fs system when used as designed, called with the compilation
> > > wordlist at the front of the search order.
>
> Indeed, if you want to make a system targeted to switching between
> private and hidden words, why not just call what you are calling
> PRIVATE, "HIDDEN", and call what you are calling PUBLIC, "PRIVATE". If
> the system is not set up to actually provide PUBLIC words, it should
> not be labeling sections *as* PUBLIC.

Bruce, I don't disagree with some of the things you are saying. But,
I'll postpone replying in detail until I can make a thoughtful
response.

Krishna

On Oct 15, 11:04=A0am, Albert van der Horst <alb...@spenarnc.xs4all.nl>
wrote:

> In article <j7bd27$q...@news-01.bur.connect.com.au>, Ed <nos...@invalid.c=
om> wrote:
> >Hidden words don't go anywhere.

>> Word-hiding works on the premise that certain words are used only
>> during creation of a library or app, after which they may be rendered
>> invisible.

>> There are various ways one can make a word invisible e.g. unlinking
>> its name or putting it in a special wordlist created for the purpose.
>> Either way, if you think you may need the word later then don't hide
>> it.

Published source libraries complicate this question, since the
original source author may in fact not need that word later, but some
other author has a use for the source library where they may.

>> Indeed, closed modules and hidden words could be seen as antithetical
>> to Forth since factoring produces many potentially useful and
>> re-usable words.

Yes, this is the conflict ~ the clear spelling out of dependencies and
the control against unintended visibility of the wrong word, for use
or when engaged in conditional compilation to avoid unnecessarily
compiling the same word twice.

> As the use of libraries is mainly the situation where you don't
> want to see the internals of modules, this can be construed as
> "library use is antithetical to Forth".

One qualification is that there is a difference between, "normally do
not want to see the internals" and "don't ever want to see the
internals of modules". Sometimes you *do* ~ as Andrew Haley points
out, when doing maintenance on the module or a user of the module.

> Maybe it is. If I look at the "library" in my Forth (.lab) it
> is tree like and it is conceived as a whole. If you want a
> branch, you pull in all branches that are used, unless it
> is already there. Whether you have a facility, is checked
> by ... looking up the word itself. If it is not there, it
> must be loaded.

Is this a single library that has already been integrated with the
system?

A step up in complexity in terms of namespace management is an
external published source library, that is integrated with itself,
but, intrinsically, not integrated with each implementation that might
make use of it.

Further complexity would be multiple such libraries, which are not
necessarily integrated with each other.

The size of the problem is substantially reduced if a portion of the
words have to be integrated. That is the fundamental division between
PUBLIC and PRIVATE in a published source library: how a PUBLIC word
will integrate into the system compiling the library "in public", has
to be thought through, but PRIVATE words only have to work in terms of
the library itself.

The siren song is a system where you load precisely the module you
wish, and it loads exactly what it needs, and nothing more, so if you
really need the whole thing, you load the whole thing, but if you only
really need three modules, you only load three modules.

Which brings dependencies into it.

> It is almost like
>
> 117 LOAD
> and in screen 117
> =A0 =A023 LOAD 111 LOAD
> =A0 =A0...
> and in screen 23
> =A0 18 4 THRU
> =A0 ...
>
> but without the screen numbers (and no double loading of
> the same word.)

No double loading of the same word is what steps up in complexity when
the library is not necessarily integrated with the namespace into
which it is compiled, since this interacts with the public/private
boundary.

I'm willing to presume that part of the public/private boundary is
intelligent use of the library given library useage documentation what
PUBLIC words are defined by each word, and what other PUBLIC words are
entailed by loading the module.

But the *point* of having PRIVATE words is that the implementer or
other user does *not* have to worry about integration of the library
PRIVATE words into their namespace. Which means that conditional
compilation *can't* be built around [UNDEFINED] since the external
context cannot be presumed to be integrated with all private library
namespaces.

One thing that seems to be a consensus part of modules under
discussion is preserving search order state by preserving the search
order depth and restoring that depth at the end of the module. Simply
adopting a library coding standard that the library never modifies the
part of the search order it did not add itself makes that workable.

Being ABLE to not redefine private words already defined ~ including
private words ~ requires that private words not be hidden. That
facility is provided, of course, whether the library compiles all
private words into their own named wordlists, the private words for a
module are accessible via a dual wordlist module structure in the
dictionary, or a variety of other solutions, and it could be a
compilation option rather than built into the system.

If there is an ability to name a wordlist, then inside the scope of
the library:
   IMPORT: privatename
.... can bring it into the search order. Having that as a parsing word
supports integration into the host namespace system, since if a
VOCABULARY is present, IMPORT: can take into account whether the
vocabulary is self-pushing or F83.

If you have the search order space you want, [UNDEFINED] is
problematic for private words, as discussed. However, the "prior-
depth" supports working out which wordlists in the search order were
there when the module started, hence is "untrusted" namespace, and
which wordlists were added by the module, and hence is "trusted"
namespace. So:
   [MODULE-LACKS] wordname [IF] ... [ELSE] ... [THEN]
.... can be restricted to the "trusted" namespace, searching any and
all wordlists in the "trusted" space. [MODULE-HAS] is the converse.

On Oct 15, 6:32=A0am, Albert van der Horst <alb...@spenarnc.xs4all.nl>
wrote:
> In article <9391aac2-bd46-4185-885b-c891b9ced...@hv4g2000vbb.googlegroups=
..com>,
>
> BruceMcF =A0<agil...@netscape.net> wrote:
>
> >That eliminates problems in the event of a name clash between a module
> >name and another definition by the same name loaded in the interim,
> >once making sure that the module name is never executed directly by
> >source library code is adopted as a library-side integration
> >requirement.

> This amounts to not using private names in MACRO's where those
> private names are used in other modules too.
> A big no no (as Anton Ertl will certainly agree to.)
> This, of course, happens minly if the library writer delays
> use till run time, in this fasion.

I didn't think of the macro issue at all, but now that you mention it,
the module system allows "safe" MACRO use that would become unsafe if
a dummy flat wordspace was in effect.

"module names" as I was using it above is not talking about private
words in the module but the name of the module itself, which is used
in dependency operations in a module.

What I was thinking of as the following issue. Consider that you have
a library designed to be loaded by the required function, which
additional support modules loaded by that first library module. And
then you load two different applications that both use that library.

Your dictionary can have:

app1 code ... library code ... app1 code ... app2 code ... more
library code ... app2 code.

What the above is talking about is at "more library code", where the
modules already loaded by app1 are re-used by app2, and only the
additional modules required by app2 appear there.

After all, two different apps both engaged in floating point
applications, both using the FSL, and the two not both using identical
modules from the FSL is certainly a plausible scenario.

Now, suppose that in the structure version, module names are defined
into MAKE-MODULE or some such vocabulary. Even if app1 or the
preliminary section of app2 uses that module name to mean something
else entirely, this is normally no problem, because the MAKE-MODULE
vocabulary is above the external context in the search order.

But what if there is a dummy definition of module, defining everything
into the original compilation wordlist and ignoring all the namespace
structure space ... as may be required to import FSL modules in a
cross-compiler context.

If the syntax of dependency declarations bringing a module into the
namespace is:
   <module-name>
.... then <module-name> is defined as a dummy and nothing happens ...
EXCEPT if there is a name clash, and then ONLY with the dummy version,
in which case, because its a flat namespace, the same name doing
something dramatically different happens.

If the syntax of dependency declarations bringing a module into the
namespace is:
   IMPORT: <module-name>
.... then the namespace conflict that arises with the dummy version is
rendered harmless, since IMPORT: in the dummy version just parses and
discards the following name.

tl;dr version: the flat namespace is prone to a namespace conflict if
it executes module names, but it doesn't *need* to execute module
names, since they are dummies in the flat namespace version. So use a
parsing search order set-up, and that eliminates a potential problem
by design.

On Oct 14, 2:37=A0pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:

> OK, so it requires a typo, not just a name clash. =A0Fair enough.

I don't know that a typo is required, but that's the example that was
given.

> Or that it does; it's your claim.

> quod gratis asseritur, gratis negatur: what is asserted without reason ma=
y be denied without reason.

Itsay Otnaty Eitheray Erehay Oray Erethay ~ Its Neither Here Nor
There.

My view is that its a substantial problem, but whether or not I can
defend my view on the issue is entirely beside the point when the FSL
is an already existing published source code library.

> It's not so much point and drool as old school. =A0Searching in the
> editor is what we did before LOCATE.

> > If its more than a click or two and a keypress or two, its not the
> > searching that is stone age so much as the source file editor.

> OK, but without knowing the dictionary search context (or perhaps
> decompiling the definition) how does the editor know where in the
> source code a word is defined? =A0There may be all manner of conditional
> compilation, words generated on the fly, and so on. =A0That's the thing
> about LOCATE: it's always right.

Yes, which tells me that a Forth source code viewer written in Forth
so that I can add a locate name function to the right click menu has
substantial appeal compared to a generic programming editor.

Also neither here nor there, I guess.

>>> I'm vaguely in favour of grouping words together, but I suppose it
> >> doesn't affect me as much as others.

>> On the other thread under this heading, I've argued there are two
>> general approaches to allowing the users of a library to retain the
>> control over the structure of their application namespace while
>> including published library code that includes a modular compilation
>> system.

>> (1) The general Swiftforth approach in "PACKAGE", where the entire
>> control of where public words are compiled belongs to the caller ~ the
>> library simply does not give a structure to the public namespace.

>> (2) A small, flexible toolkit that allows the library to to put the
>> default public word structure in one file that includes the underlying
>> module definitions, so that a different loadscript, or no loadscript
>> at all, can give a different structure.

>> I'm thinking that (2) addresses your concern most directly for an
>> application that loads the modules themselves, since it would allow
>> just setting up a single namespace for all private words using that
>> modular system.

> Right.

> > I could sketch more seamless and complicated integrations, but in
> > any event, in the same loadscript define a word ("get-private") to
> > bring that namespace into the search order and bob's your uncle.

> Sounds good.

Right now I'm thinking of the following:

(1) The main module system has the module system, with default
dependency and search order set-up in the base module definition, but
the default can be overridden prior to loading.

(1.1) Modules can be multi-headed, and the first header word
determines the module system in use. This allows a user to either
accept the default model built into library models, or impose its own
model by executing its own module header before including the module
file.

(1.2) This extends to dependency management ~ the search order
operations following the first header word are active, the search
order operations following subsequent header words are inactive. A
user or alternative loadscript the imposes their own module model
accepts responsibility for dependency management.

(1.2) Module-Name: At the head of the module, saves system state and
creates a wordlist in the modules system vocabulary for the module
PUBLIC words, setting up a name for the entire module, making it the
current compilation wordlist. Otherwise after the head of the module,
turn off the following search order words.

(1.3) Anonymous-Module At the head of the module, saves system state,
and set up the current compilation wordlist as the PUBLIC wordlist.
Otherwise, turn off the following search order words.

(1.4) Private-Name: At the head of the module, saves system state and
creates or reuses a private words vocabulary in the modules system
vocabulary for the module PRIVATE words. Following Private-name, a no-
op. After the head of the module, turn off the following search order
words and, if the original header was Module-Name:, find or create the
name and store it as the private wordlist in the Module-Name:
structure.

(1.5) Begin-Module saves system state if at the head of the module.
Create an anonymous PRIVATE wordlist if a private wordlist does not
already exist. Set the current compilation wordlist as the PUBLIC
compilation wordlist. Turn on search order words. Set the compilation
wordlist to PRIVATE.

(2) Two simpler alternate versions of the module system, which would
be compiled in place of the full fledged system.
(2.1) Alternate 1 is a pure flat system, with all words defined into
and visible in the original compilation wordlist.
(2.2) Alternate 2 has a single module-system-level vocabulary, with
all PUBLIC words in the original compilation directory and all PRIVATE
words in the module-system-level vocabulary.

So 2.2 would be something like:


[UNDEFINED] VOCABULARY [IF] \ Forth83 vocabulary if not here
   : VOCABULARY ( "name" - ) \ "name" ( O: wid0 -- wid-"name" )
      WORDLIST CREATE ,
      DOES> @ >R GET-ORDER NIP R> SWAP SET-ORDER ;
[THEN]
[UNDEFINED] ALSO [IF] \ A Forth94 standard word
   : ALSO ( -- ) GET-ORDER OVER SWAP 1+ SET-ORDER ;
[THEN]

VOCABULARY Make-Module

GET-CURRENT ALSO Make-Module DEFINITIONS
   \ Don't yet know if F83 or self-pushing vocab

VARIABLE public-wid
( original-compilation-wid ) public-wid !

[UNDEFINED] order> [IF]
   : order> ( -- wid ) ( O: wid -- )
      GET-ORDER SWAP >R 1- SET-ORDER R> ;

[UNDEFINED] order@ [IF]
   : order@ ( -- wid ) ( O: wid -- wid )
      GET-ORDER OVER >R drops R> ;

order@ CONSTANT Make-Module-wid

\ ***********************************************
\ Find out which kind of vocabulary and
\ define the correct host-also to accommodate it.

Make-Module order> order@ Make-Module-wid =3D [IF]
   \ self-pushing vocabulary, discard redundant
   DROP

   : host-also ( O: -- ) ; IMMEDIATE
[ELSE]
   \ F83 vocabulary, restore wordlist
   >order

   : host-also ( O: wid -- wid wid ) ALSO ;
[THEN]
\ ***********************************************

\ PUBLIC PRIVATE and EXPORT are only used
\ between Begin-Module and End-Module

: PUBLIC ( -- ) public-wid @ SET-CURRENT ;
: PRIVATE ( -- ) Make-Module-wid SET-CURRENT ;

: Import: ( "name" -- ) PARSE-NAME 2DROP ;
: Importall: ( "name" -- ) PARSE-NAME 2DROP ;

: End-Module ( -- ) PUBLIC PREVIOUS 0 public-wid ! ;

\ This system imposes the same module model no matter what
\ The first header encountered does a Begin-Module
\ saving current compilation wordlist as public-wid.

: ?first-header ( -- )
   public-wid @ 0=3D IF
      GET-CURRENT public-wid !
      host-also Make-Module DEFINITIONS
   THEN ;

PUBLIC

: Begin-Module ( -- ) ?first-header
: Anonymous-Module ( -- ) ?first-header ;
: Module-Name: ( "name" -- ) ?first-header PARSE-NAME 2DROP ;
: Private-Name: ( "name" -- ) ?first-header PARSE-NAME 2DROP ;

On 10/15/11 4:25 PM, BruceMcF wrote:
> On Oct 14, 2:37 pm, Andrew Haley<andre...@littlepinkcloud.invalid>
> wrote:


>>> If its more than a click or two and a keypress or two, its not the
>>> searching that is stone age so much as the source file editor.
>
>> OK, but without knowing the dictionary search context (or perhaps
>> decompiling the definition) how does the editor know where in the
>> source code a word is defined?  There may be all manner of conditional
>> compilation, words generated on the fly, and so on.  That's the thing
>> about LOCATE: it's always right.
>
> Yes, which tells me that a Forth source code viewer written in Forth
> so that I can add a locate name function to the right click menu has
> substantial appeal compared to a generic programming editor.

I agree that being able to instantly summon a definition as you suggest 
"from within the editor" is a great way to go.  One can then do the same 
for words that make up the definition and so on.  The key is two-way 
communication between the editor and the Forth system.

-Doug

On Oct 8, 11:42=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> Here's a better factoring, which makes clear that MODULE: only
> provides a naming mechanism. The wordlist creation and other
> initialization is now entirely within BEGIN-MODULE.

OK, here are my suggestions:

(1) Inside the module, the search order should not be modified by
vocabularies executing on their own. Rather, there should be parsing
words that do that.

Advantage: Allows use of host vocabulary whether it is self-pushing or
F83 semantics. Allows multi-headed modules. Allows dummy
implementation with less name clash integration to be done by hand.

(1.1) Location of module names

Less cluttering of the user namespace points toward storing in their
dedicated wordlist. There has to be a wordlist for PUBLIC/PRIVATE to
av avoid interference with other PRIVATE/PUBLIC implementations in the
host, so that can be used and then placing it in the search order
ensures that both the module operators.

I've been tentatively calling it Make-Module

(1.2) Proposed operations:

Import-module: <name>
 Puts the named module (in module system wordlist) in the search
order.
Import-all: <name>
 Puts the named module and its private module in the search order.
External: <name>
 Puts the external vocabulary in the search order.
Export-to: <name>
 Sets up a wordlist as a compilation target. Creates it as a
vocabulary in the current compilation wordlist if it is not presently
visible.

(2) Multi-headed Modules

Modules always have a single unique Begin-Module, but they can
optionally be initiated by other module initiating words. I am calling
initiators other than Begin-Module module header words. The section
between and including one module header word and either another module
header word or Begin-Module is a single header.

In multi-heading module definitions, the earlier module header and and
often does pre-empt a later module header. This may include turning
off the search order modifying words above in a pre-empted header.

This allows a library to simply define the module with the module
model it likes, and if an implementation or user prefers a different
module model, they define a loadscript that sets up the module model
*they* want, and then call the module definition. The loadscript pre-
empts or modifies the behavior of the following header as desired.

Note that this is not helter skelter mix-and-max ~ pre-empting the
module model used by the library intrinsically implies pre-empting its
dependency management, so the loadscript has also got to be sure that
dependency management is handled correctly for its model.

(2.1) The named PUBLIC/PRIVATE pair model.

In this model, "Module-Name: <name>" creates a structure to hold the
module information, stores the search order state, brings the module
system wordlist into the search order, creates a public wordlist,
stores it in the module structure, and brings it into the search order
as the current compilation target. The target location for the private
search order is stored in a module system variable for later use. If
Begin-Module is reached without a private-wid being created, it
creates one and stores it at the location set by Module-Name:.

If pre-empted, this header does nothing.

If there is a following "Private-name:" header, the first one
encountered creates or re-uses a named private vocabulary in the
module system vocabulary, and stores it in the private wid section of
the named module structure. Otherwise, Begin-Module creates and stores
the private wordlist in the named module structure.

(2.2) The named PRIVATE model.

In this model, "Private-Name: <private-vocab>" either creates or
reuses a private vocabulary in the module system vocabulary, the prior
CURRENT wordlist is the PUBLIC wordlist, the search order state is
stored, and it is ensured that CURRENT is on top of the search order,
then Make-Module.

If a Private-Name: header follows a Module-name: header, see above.
Otherwise, if Private-name: is pre-empted, it discards its name and
turns off the following search order words in its header.

(2.3) The Anonymous-Module model

"Anonymous-Module" saves the search order state, sets the public-wid
to the CURRENT wid, and sets the private wid store variable to 0. When
Begin-Module is reached, it created a private-wid This results in the
default model that occurs with a Begin-Module alone.

If preempted, it turns off the following search order words in its
header.

(2.4) Module Header detection.

"prior-depth" must be set by any header. It it set to 0 by system
initialization and by End-Module (after resetting to the stored
depth). Therefore, the first header to execute finds a 0 prior-depth,
and stores the current search order depth in it while saving search
order state.

"header-off" is set to 0 by system initialization and by Begin-Module.
A header that is preempted and is turning its dependency management
operations off will set this to true.

"store-private" is a variable that is set to 0 by system
initialization and End-Module. It discriminates between the Module-
name: model and the other two.

For Anonymous-Module, the conditional execution is:
   prior-depth=3D0? Perform
   prior-depth<>0? Set header-off to TRUE

For Module-Name:, the conditional execution is:
   prior-depth=3D0? Perform
   prior-depth<>0? Discard <name> and set header-off to TRUE

The special case is "Private-Name: <name>":
   prior-depth=3D0? Perform
   prior-depth<>0? set header-off to TRUE:
      public-wid<>prior-current AND private-wid=3D0:
         Perform Module-name: part of Private-Name:
      ELSE Discard <name>

There's an implementation dependency on no wid=3D0, but if that were an
issue, then that's implementation version could overload that state
info as the sign of header-off.

(3) Combining search order operations and model after Begin-Module

PUBLIC and PRIVATE are set after Begin-Module completes.
"External: <name>", "Export-to: <name>" and EXPORT are oriented to
vocabularies outside the module system, so they are indifferent to
module model in force.

"Import-Module: <name>" is only meaningful under the Module-Name:
model ~ public-wid<>prior-current detects that model. Otherwise it
discards the name token.

"Import-All: <name>" needs to add the module wid under the Module-
Name: and add the private wid under both Module-Name: and Private-
Name:. Under Anonymous modules, it discards the name token.

Since the named module structure emulates a host vocabulary, Import-
All: goes into the word only if it is a named module and adds the
private order, then it executes the named word as a host vocabulary,
therefore bringing in the public wordlist under Module-Name: and the
private wordlist under Private-Name:.

On 10/15/11 12:30 PM, Doug Hoffman wrote:
> On 10/15/11 4:25 PM, BruceMcF wrote:
>> On Oct 14, 2:37 pm, Andrew Haley<andre...@littlepinkcloud.invalid>
>> wrote:
>
>
>>>> If its more than a click or two and a keypress or two, its not the
>>>> searching that is stone age so much as the source file editor.
>>
>>> OK, but without knowing the dictionary search context (or perhaps
>>> decompiling the definition) how does the editor know where in the
>>> source code a word is defined? There may be all manner of conditional
>>> compilation, words generated on the fly, and so on. That's the thing
>>> about LOCATE: it's always right.
>>
>> Yes, which tells me that a Forth source code viewer written in Forth
>> so that I can add a locate name function to the right click menu has
>> substantial appeal compared to a generic programming editor.
>
> I agree that being able to instantly summon a definition as you suggest
> "from within the editor" is a great way to go. One can then do the same
> for words that make up the definition and so on. The key is two-way
> communication between the editor and the Forth system.

You can do all this within the command window of SwiftForth, even if 
you're using an external editor.  LOCATE will display not only the 
definition but also some source before and after, so you can see 
context.  Also (from the SwiftForth manual):

"If you select a word in the command window, the popup menu generated by 
a rightclick will offer (among others) the option Edit. Selecting that 
command will launch your linked editor (or switch to it if it is already 
open) with the cursor positioned on the source line containing the word. 
This feature lets you immediately edit the source, if you wish, and 
examine the rest of the file."

The full list of right-click options includes LOCATE, EDIT, SEE 
(disassemble), cross-reference, execute, copy, and paste (whatever's in 
the clipboard into that point in the command window).

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Oct 15, 7:28=A0pm, "Elizabeth D. Rather" <erat...@forth.com> wrote:
> You can do all this within the command window of SwiftForth, even if
> you're using an external editor. =A0LOCATE will display not only the
> definition but also some source before and after, so you can see
> context.

Yeah, SwiftForth, fr'instance.

On 10/15/11 7:28 PM, Elizabeth D. Rather wrote:
> On 10/15/11 12:30 PM, Doug Hoffman wrote:
>> On 10/15/11 4:25 PM, BruceMcF wrote:
>>> On Oct 14, 2:37 pm, Andrew Haley<andre...@littlepinkcloud.invalid>
>>> wrote:
>>
>>
>>>>> If its more than a click or two and a keypress or two, its not the
>>>>> searching that is stone age so much as the source file editor.
>>>
>>>> OK, but without knowing the dictionary search context (or perhaps
>>>> decompiling the definition) how does the editor know where in the
>>>> source code a word is defined? There may be all manner of conditional
>>>> compilation, words generated on the fly, and so on. That's the thing
>>>> about LOCATE: it's always right.
>>>
>>> Yes, which tells me that a Forth source code viewer written in Forth
>>> so that I can add a locate name function to the right click menu has
>>> substantial appeal compared to a generic programming editor.
>>
>> I agree that being able to instantly summon a definition as you suggest
>> "from within the editor" is a great way to go. One can then do the same
>> for words that make up the definition and so on. The key is two-way
>> communication between the editor and the Forth system.
>
> You can do all this within the command window of SwiftForth, even if
> you're using an external editor. LOCATE will display not only the
> definition but also some source before and after, so you can see
> context.

Yes.  SwiftForth is very good about this it seems.  Of course that is 
different from doing it completely from within an editor but that may be 
nitpicking.  Different people will have different preferences for 
editor/Forth-system interaction.

-Doug

On 10/15/11 4:46 PM, Doug Hoffman wrote:
....
>>> I agree that being able to instantly summon a definition as you suggest
>>> "from within the editor" is a great way to go. One can then do the same
>>> for words that make up the definition and so on. The key is two-way
>>> communication between the editor and the Forth system.
>>
>> You can do all this within the command window of SwiftForth, even if
>> you're using an external editor. LOCATE will display not only the
>> definition but also some source before and after, so you can see
>> context.
>
> Yes. SwiftForth is very good about this it seems. Of course that is
> different from doing it completely from within an editor but that may be
> nitpicking. Different people will have different preferences for
> editor/Forth-system interaction.

The work style that SwiftForth is aimed at is doing all your testing in 
the command window, and only going to the editor to make changes in a 
file. Using the editor as the primary work platform is probably common 
with less inherently interactive languages.

And, as SwiftForth can work on the program as loaded, it not only won't 
be confused about different versions of a word, but will also do things 
like cross-references based on the running code, rather than a file 
search (which can return instances that aren't being used).

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Oct 15, 11:50=A0pm, "Ed" <nos...@invalid.com> wrote:
> I suggest that in Forth there is no compiled 'block of code'. =A0What
> one has is a collection of words that have been added to the dictionary.
> Some will be 'end-user' words, and the rest, words which may be
> re-used later. =A0Hiding prevents re-use which is why Forth has never
> encouraged it.

OTOH, in a large enough namespace, organization aids in finding what
already exists, thus encouraging re-use.

I'm a fan of hierarchies, but shallow and wide ones. That's one reason
why I don't want a hundred module wordlists with an average of four to
each words each.

Krishna Myneni wrote:
> On Oct 15, 3:36 am, "Ed" <nos...@invalid.com> wrote:
> > jacko wrote:
> > > Where do the hidden words go? From the point of view of the adapter, surely they must remain visible
> > > somewhere.
> >
> > Hidden words don't go anywhere.
> >
> > Word-hiding works on the premise that certain words are used only
> > during creation of a library or app, after which they may be rendered
> > invisible.
> >
> > There are various ways one can make a word invisible e.g. unlinking
> > its name or putting it in a special wordlist created for the purpose.
> > Either way, if you think you may need the word later then don't hide it.
> >
>
> Sure, and the modular programming code we have published is one way of
> doing that. If that seems excessively complex to you, by all means,
> use a simpler approach.

Yes, but Forth is about simplicity - finding the most direct and efficient
way of achieving the desired goal.  Unlinking a word header or smudging
it is undoubtedly the most direct way of rendering it invisible.   If one can't
do that then work-arounds using wordlists may be the only option.

> > Indeed, closed modules and hidden words could be seen as antithetical
> > to Forth since factoring produces many potentially useful and re-usable
> > words.
>
> In the real world, programmers sometimes have to interface to things
> outside of the Forth environment. Let's take a concrete example:
>
> ftp://ccreweb.org/software/kforth/examples/gpib.4th
>
> The above is an interface to an external device driver, written in C,
> for a General Purpose Interface Bus. It is impractical to consider
> coding the actual device driver in Forth (using the Forth's assembler,
> for example), and then one would have to operate with root privileges
> -- of course you may want to write your own Forth operating system and
> use that. By all means, do so.

I'm not sure what you mean.  I've written a PC parallel port interface
for a forth-based eprom programmer.  There was no need or benefit
in hiding the details from the application.  I also wrote a graphics
library to interface to a Borland BGI driver.  In that I did hide many of
the low level words - mostly for cosmetic reasons (we all like to hide
our mess :)

> But, going back to the above example, it is clear that the driver-
> dependent constants are used only by the interface words. There's no
> need for constants like these to be visible to the user and clutter
> the search order, *once the interface is debugged*.

Windows-based forths effectively have thousands of API constants
in the search order.  No-one complains of clutter there.

> The concept of a
> module is that it is a reusable block of code with a public interface
> that has been tested.

I suggest that in Forth there is no compiled 'block of code'.  What
one has is a collection of words that have been added to the dictionary.
Some will be 'end-user' words, and the rest, words which may be
re-used later.  Hiding prevents re-use which is why Forth has never
encouraged it.

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> In the real world, programmers sometimes have to interface to things
> outside of the Forth environment. Let's take a concrete example:
> 
> ftp://ccreweb.org/software/kforth/examples/gpib.4th
> 
> The above is an interface to an external device driver, written in C,
> for a General Purpose Interface Bus. It is impractical to consider
> coding the actual device driver in Forth (using the Forth's assembler,
> for example), and then one would have to operate with root privileges
> -- of course you may want to write your own Forth operating system and
> use that. By all means, do so.
> 
> But, going back to the above example, it is clear that the driver-
> dependent constants are used only by the interface words. There's no
> need for constants like these to be visible to the user and clutter
> the search order, *once the interface is debugged*.

What harm do those constants do, though?  I suppose they might very
slightly slow down dictionary searches, but I can't think of anything
else.  So why dows it matter?

Andrew,

On 10/15/11 11:33 PM, Elizabeth D. Rather wrote:
> On 10/15/11 4:46 PM, Doug Hoffman wrote:
> ...
>>>> I agree that being able to instantly summon a definition as you suggest
>>>> "from within the editor" is a great way to go. One can then do the same
>>>> for words that make up the definition and so on. The key is two-way
>>>> communication between the editor and the Forth system.
>>>
>>> You can do all this within the command window of SwiftForth, even if
>>> you're using an external editor. LOCATE will display not only the
>>> definition but also some source before and after, so you can see
>>> context.
>>
>> Yes. SwiftForth is very good about this it seems. Of course that is
>> different from doing it completely from within an editor but that may be
>> nitpicking. Different people will have different preferences for
>> editor/Forth-system interaction.
>
> The work style that SwiftForth is aimed at is doing all your testing in
> the command window, and only going to the editor to make changes in a
> file.

Understood.  But, when an editor window *is* a console window then that 
changes things.  This requires 2-way communication between the editor 
and the Forth system, instead of just 1-way communication Forth-> 
editor.  There are Forths/editors that do this.

> And, as SwiftForth can work on the program as loaded, it not only won't
> be confused about different versions of a word, but will also do things
> like cross-references based on the running code, rather than a file
> search (which can return instances that aren't being used).

With 2-way communication the correct source file is always opened from 
the editor.  I'm not knocking switching back and forth between the Forth 
application and the editor application.  That works fine.  In some 
Forths I've seen the console has extra behaviors to try to make the 
console work somewhat like an editor: such as scrolling back into a 
history buffer and being able to copy/paste from that.  Some Forths 
simply go all the way and allow the editor window(s) to *be* the 
console(s).  There is no wrong/right way to do these things, just 
different ways.

-Doug

Albert van der Horst wrote:
> In article <j7bd27$qu$1@news-01.bur.connect.com.au>,
> Ed <nospam@invalid.com> wrote:
> ...
> >Indeed, closed modules and hidden words could be seen as antithetical
> >to Forth since factoring produces many potentially useful and re-usable
> >words.
>
> As the use of libraries is mainly the situation where you don't
> want to see the internals of modules, this can be construed as
> "library use is antithetical to Forth".
> ...

I'm not saying that.  I use libraries :)

What I question is the premise that making certain words nameless
or hiding them in a module will somehow make applications easier
to write.

If we were taking binary/object modules then of course only the
end-user words should be made public.  However forth works at the
source level and there's no necessity to hide/strip names.

Andrew Haley <andrew29@littlepinkcloud.invalid> writes:
>That's the thing
>about LOCATE: it's always right.

Hmm?  In your example, where you decompile a word, and then LOCATE the
words it calls, there is no guarantee at all that LOCATE will show you
the called word.  I think that was your complaint about the proposed
module system, no?

But even without any wordlists etc. LOCATE can show the wrong word:

: foo ... ;
: bar ... foo ... ;
: foo ... ;

"SEE BAR" will show you a call to the first FOO, but LOCATE FOO (as
well as SEE FOO) will show you the second FOO.

A well-integrated editor (that gets a lot of info from the Forth
system) can show you the right FOO when you click on the FOO in the
definition of BAR.

A not so well integrated editor (e.g., Emacs with Gforth's tags) will
show you one of the FOOs, and give you the option of also seeing all
the others.

LOCATE just shows you the last FOO, no other options that I know of.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:
> Andrew Haley <andrew29@littlepinkcloud.invalid> writes:
>>That's the thing about LOCATE: it's always right.
> 
> Hmm?  In your example, where you decompile a word, and then LOCATE the
> words it calls, there is no guarantee at all that LOCATE will show you
> the called word.  I think that was your complaint about the proposed
> module system, no?

Not quite.  In that system, LOCATE may be not able to find a word,
even if it was not redefined.  That's a much more serious objection,
IMO: you'd be replacing a facility that sometimes fails with one
that's guaranteed to.

> But even without any wordlists etc. LOCATE can show the wrong word:
> 
> : foo ... ;
> : bar ... foo ... ;
> : foo ... ;

Of course.  Redfeinitions hide words.

> "SEE BAR" will show you a call to the first FOO, but LOCATE FOO (as
> well as SEE FOO) will show you the second FOO.

It's still right, in the current context.  No matter how a word is
created, LOCATE records the position in the input source at that
point.  That's its job.

> A well-integrated editor (that gets a lot of info from the Forth
> system) can show you the right FOO when you click on the FOO in the
> definition of BAR.

Indeed, that could be done.  Whether it should be done is another
matter.

Andrew.

On Oct 16, 12:52=A0pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:

> > A well-integrated editor (that gets a lot of info from the Forth
> > system) can show you the right FOO when you click on the FOO in the
> > definition of BAR.

> Indeed, that could be done. =A0Whether it should be done is another
> matter.

Why *shouldn't* it be done? Since such an editor can also show the FOO
presently in context, by *using* LOCATE, what is lost?

For a truly well integrated editor, some form of "flocate" could
surely be written, if not already available, so point and click versus
command line entry is not a discriminator against this kind of editor,
but rather a user preference on how to use it.

BruceMcF <agila61@netscape.net> wrote:
> On Oct 16, 12:52?pm, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>> Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> 
>> > A well-integrated editor (that gets a lot of info from the Forth
>> > system) can show you the right FOO when you click on the FOO in the
>> > definition of BAR.
> 
>> Indeed, that could be done. Whether it should be done is another
>> matter.
> 
> Why *shouldn't* it be done?

Because it potentially adds another layer of complexity.  One of
Forth's virtues, I'd argue its greatest virtue, is that very little
goes on "under the hood".  Great power comes from its simplicity and
transparency.

> Since such an editor can also show the FOO presently in context, by
> *using* LOCATE, what is lost?

If that's all it does, without parallel data structures and
complexity, all well and good.

> For a truly well integrated editor, some form of "flocate" could
> surely be written, if not already available, so point and click versus
> command line entry is not a discriminator against this kind of editor,
> but rather a user preference on how to use it.

I think that's reasonable.

Andrew.

On 10/16/11 11:38 AM, Anton Ertl wrote:

> But even without any wordlists etc. LOCATE can show the wrong word:
>
> : foo ... ;
> : bar ... foo ... ;
> : foo ... ;
>
> "SEE BAR" will show you a call to the first FOO, but LOCATE FOO (as
> well as SEE FOO) will show you the second FOO.
>
> A well-integrated editor (that gets a lot of info from the Forth
> system) can show you the right FOO when you click on the FOO in the
> definition of BAR.

That would be a very nice feature to have.

-Doug

Doug Hoffman <glidedog@gmail.com> writes:
>> A well-integrated editor (that gets a lot of info from the Forth
>> system) can show you the right FOO when you click on the FOO in the
>> definition of BAR.
>
>That would be a very nice feature to have.

The info is pretty easy to generate from the Forth system: For every
text interpretation from a file, store the position of the use and the
the position of the definition (i.e., the LOCATE info) of the found
word.  I don't know how easy it is to get text editors to make use of
that info.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

On Oct 17, 9:02=A0am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
wrote:
> Doug Hoffman <glide...@gmail.com> writes:

>>> A well-integrated editor (that gets a lot of info from the Forth
>>> system) can show you the right FOO when you click on the FOO in the
>>> definition of BAR.

>> That would be a very nice feature to have.

> The info is pretty easy to generate from the Forth system: For every
> text interpretation from a file, store the position of the use and the
> the position of the definition (i.e., the LOCATE info) of the found
> word. =A0I don't know how easy it is to get text editors to make use of
> that info.

*each* found word? Would it suffice to extend the LOCATE info
structure to include the current heads of the vocabularies in the
current search order, and an extended LOCATE, eg "LOOKIN" that does
its search based on that structure? An editor that can call LOCATE
could be set up to call LOOKIN as well.

On 10/17/11 9:02 AM, Anton Ertl wrote:
> Doug Hoffman<glidedog@gmail.com>  writes:
>>> A well-integrated editor (that gets a lot of info from the Forth
>>> system) can show you the right FOO when you click on the FOO in the
>>> definition of BAR.
>>
>> That would be a very nice feature to have.
>
> The info is pretty easy to generate from the Forth system: For every
> text interpretation from a file, store the position of the use and the
> the position of the definition (i.e., the LOCATE info) of the found
> word.

OK.  If that can somehow be done in the Forth then the ease of 
controlling the Editor will depend on a few things.  See below.

> I don't know how easy it is to get text editors to make use of
> that info.

It's not hard if one has complete control of the Editor.  If the Editor 
is part of the Forth, i.e. the same application, then this is a given. 
Having the source for a high quality Editor is the hard part.  If the 
Editor and the Forth are separate applications then one uses the 
communication protocol for that OS.  In Mops we have been using 
AppleEvents with the MacForth-based Editor for many years (2-way comm). 
  Once the 2-way comm is established there are essentially no limits to 
what can be done.  The Editor can have almost complete control of the 
Forth and vice versa.  There is an advantage in keeping the applications 
separate because if the Forth crashes it doesn't take the Editor with it 
(and so one does not lose all of the open Editor windows/files and their 
positioning on screen and so forth).  Although a fast "editor state 
restore" might make this a non-issue.

It appears that 1-way comm, Forth->Editor, is already there for a few 
basic functions on several existing quality Editors.  I suppose one 
would have to weigh the effort of expanding that and possibly making it 
2-way comm against the benefits.  The benefits may not be obvious until 
you've tried it.  One quickly gets spoiled by not having to constantly 
switch applications while programming.  The results of an interactive 
programming session can simply be left embedded in the source (commented 
out so the file will load normally).  Some Forths already have very 
Editor-like behaviors built into the console, diminishing the added 
value of 2-way comm.

-Doug

In article <2011Oct16.173806@mips.complang.tuwien.ac.at>,
Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:
>
>LOCATE just shows you the last FOO, no other options that I know of.

LOCATE is not a standard word. In ciforth e.g. it uses a source field
and it only works on words you have compiled yourself on top of an
interactive system. 1)
In that case the source is always in memory and the source of
the word can be easily shown.

1) I don't bother much,
: INCLUDED GET-FILE EVALUATE ;   modulo CATCH ing.

>
>- anton

Groetjes Albert

--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

On Oct 17, 7:03=A0am, BruceMcF <agil...@netscape.net> wrote:
> On Oct 17, 9:02=A0am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
> wrote:
>
> > Doug Hoffman <glide...@gmail.com> writes:
> >>> A well-integrated editor (that gets a lot of info from the Forth
> >>> system) can show you the right FOO when you click on the FOO in the
> >>> definition of BAR.
> >> That would be a very nice feature to have.
> > The info is pretty easy to generate from the Forth system: For every
> > text interpretation from a file, store the position of the use and the
> > the position of the definition (i.e., the LOCATE info) of the found
> > word. =A0I don't know how easy it is to get text editors to make use of
> > that info.
>
> *each* found word? Would it suffice to extend the LOCATE info
> structure to include the current heads of the vocabularies in the
> current search order, and an extended LOCATE, eg "LOOKIN" that does
> its search based on that structure? An editor that can call LOCATE
> could be set up to call LOOKIN as well.

On each open for INCLUDE add to a list the filename and store the
current HERE (or at least the addresses of where you are generating
code for this definition, and marked with [ in the example below).

Generate the header for the definition and store the line number only
in the header; no file pointer is required. On close of the INCLUDEd
file, add to the filename definition the ending HERE (marked as ] ).

For example, for a file A which includes files B and C, the code
addresses marked with [ ] might be

[----------------- file A -----------------------]
       [----- file B -----]    [- file C -]
  X                     Y                         Z

X is defined in A; Y is defined in B; Z is from the console.

Given Y, find its header and fetch the line number and code address.
Chase the files list and their range information looking for the
smallest span that contains the code address. Y is in both file B and
by inclusion in file A, but B's span is smaller. Z was defined from
the console, which has a dummy range that covers all of addressable
memory.

I've implemented this scheme with an overhead of 2 bytes per xt for
the line number (more than enough at 64K lines), which is all that's
required to do a LOCATE, along with 40 or so file definitions & a 2
cell range. The editor (context in my case) is invoked with the
filename and a parameter /g:<line number>.

On Oct 17, 3:18=A0pm, Alex McDonald <b...@rivadpm.com> wrote:
> On each open for INCLUDE add to a list the filename and store the
> current HERE (or at least the addresses of where you are generating
> code for this definition, and marked with [ in the example below).

> Generate the header for the definition and store the line number only
> in the header; no file pointer is required. On close of the INCLUDEd
> file, add to the filename definition the ending HERE (marked as ] ).
....

> I've implemented this scheme with an overhead of 2 bytes per xt for
> the line number (more than enough at 64K lines), which is all that's
> required to do a LOCATE, along with 40 or so file definitions & a 2
> cell range. The editor (context in my case) is invoked with the
> filename and a parameter /g:<line number>.

So to add the "arbitrarily never lost", all that is required is to
embed in the dictionary a search order state linked list. The location
of the header currently located is used to infer the search order
state, with its own location giving the head of the chain to search
for the current compilation wordlist in its prevailing search order.

OTOH, in a well organized module system, if the PUBLIC words can be
located, then under the above their original source file can be found,
and the private words are in there. So given the filename already from
the above, the "arbitrarily never lost" extension is overkill ~ it
only gives a substantial gain in usability for the quasi-public words
stored in their own wordlists.

On 10/17/11 3:02 AM, Anton Ertl wrote:
> Doug Hoffman<glidedog@gmail.com>  writes:
>>> A well-integrated editor (that gets a lot of info from the Forth
>>> system) can show you the right FOO when you click on the FOO in the
>>> definition of BAR.
>>
>> That would be a very nice feature to have.
>
> The info is pretty easy to generate from the Forth system: For every
> text interpretation from a file, store the position of the use and the
> the position of the definition (i.e., the LOCATE info) of the found
> word.  I don't know how easy it is to get text editors to make use of
> that info.

Exactly.  That is, in fact, how SwiftForth's right-click features work. 
If you right-click a word in the command window, it will show you the 
options for *that* word. If it's in a definition displayed in the 
command window (say, from a prior LOCATE) it will apply its options to 
*that* version of that word, even if it has been subsequently redefined.

Much harder to do this from a separate editor without building a whole 
parallel infrastructure.

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Oct 17, 1:27=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 17, 3:18=A0pm, Alex McDonald <b...@rivadpm.com> wrote:
>
> > On each open for INCLUDE add to a list the filename and store the
> > current HERE (or at least the addresses of where you are generating
> > code for this definition, and marked with [ in the example below).
> > Generate the header for the definition and store the line number only
> > in the header; no file pointer is required. On close of the INCLUDEd
> > file, add to the filename definition the ending HERE (marked as ] ).
>
> ...
>
> > I've implemented this scheme with an overhead of 2 bytes per xt for
> > the line number (more than enough at 64K lines), which is all that's
> > required to do a LOCATE, along with 40 or so file definitions & a 2
> > cell range. The editor (context in my case) is invoked with the
> > filename and a parameter /g:<line number>.
>
> So to add the "arbitrarily never lost", all that is required is to
> embed in the dictionary a search order state linked list. The location
> of the header currently located is used to infer the search order
> state, with its own location giving the head of the chain to search
> for the current compilation wordlist in its prevailing search order.
>
> OTOH, in a well organized module system, if the PUBLIC words can be
> located, then under the above their original source file can be found,
> and the private words are in there. So given the filename already from
> the above, the "arbitrarily never lost" extension is overkill ~ it
> only gives a substantial gain in usability for the quasi-public words
> stored in their own wordlists.

I also have a "super-locate" that provides the location of every named
XT, hidden, duplicated or otherwise, along with a number that can be
used to refine the edit. EDIT just edits the FINDable word. As per the
following (human stuff in upper case);

INCLUDE TEMP1  ok
INCLUDE TEMP2  ok
EDIT HIDDENXT
             ^
Error -13 HIDDENXT is undefined
WHERE HIDDENXT
1 in src\temp1.f at line 12
2 in src\temp2.f at line 3  ok
2 NEDIT HIDDENXT  in src\temp2.f at line 3  ok

An extension I am toying with, since it's a small amount of code to
implement given the vocabulary and header structure, is the word that
finds the filename and the line number for an XT be extended to return
the vocabulary/wordlist in which it's defined.

One other cool feature (in development, and it only partly works at
the moment); if a word was defined from the console, then [N]EDITing
it opens the editor up on a file that contains a copy of the console
buffer. It's read only, but can be used for cut&paste for editors that
use multiple tabs or windows.

BruceMcF <agila61@netscape.net> writes:
>On Oct 17, 9:02=A0am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
>wrote:
>> Doug Hoffman <glide...@gmail.com> writes:
>
>>>> A well-integrated editor (that gets a lot of info from the Forth
>>>> system) can show you the right FOO when you click on the FOO in the
>>>> definition of BAR.
>
>>> That would be a very nice feature to have.
>
>> The info is pretty easy to generate from the Forth system: For every
>> text interpretation from a file, store the position of the use and the
>> the position of the definition (i.e., the LOCATE info) of the found
>> word. =A0I don't know how easy it is to get text editors to make use of
>> that info.
>
>*each* found word? Would it suffice to extend the LOCATE info
>structure to include the current heads of the vocabularies in the
>current search order, and an extended LOCATE, eg "LOOKIN" that does
>its search based on that structure? An editor that can call LOCATE
>could be set up to call LOOKIN as well.

Sure, other implementation strategies are possible.  The one I
suggested would be simple and needs only one-way communication from
the Forth system to the editor.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:

> What should modules achieve.

I feel modules are useful mainly because they help solve the problem of 
name clashes when writing and using libraries. 

For example: 

module: Foo
begin-module

: say ." Foo " ;

end-module

module: Bar
begin-module

: say ." Bar " ;

end-module

\ Used in a named or anonymous module
begin-module
Foo \ Depends on Foo
Bar \ Depends on Bar

: say ." My " Foo.say Bar.say

end-module

OR if there were no ambiguity about what "say" was: 

: say2 ." my " say ;  

(Of course, I'm not insisting on the Java "." syntax.) 

Is this practicable? Are there better alternatives? 

When using see/locate, having to include the module's name as a prefix 
(ie. Foo.bar) in the REPL is easier if there are few name conflicts, 
since the prefix would not be required if the name is unique. 
If there are many name conflicts, "opening/closing" the module from the 
REPL to fix the search order is better.

Perhaps a mix of both strategies?

Of course, both see/locate etc. would have to be retrofitted by the 
module system to do the sensible thing if a member resides within a 
module. For example, the system should respond with Foo's "say" if the 
user asks for a listing of "say" in my second usage example. If both Foo 
and Bar were loaded, then the system prompts the user to specify
which of Foo.say or Bar.say is his intention. 

A problem may often arise using see/locate when two modules import 
functions of the same name from other different modules. The obvious (and 
best, IMO) solution is for the module system to automatically prefix each 
member with the module's name. This may not be palatable to everyone. 
Another "solution" is a function called "modules?" that lists the 
dependencies of each function. For example, "say modules?" would respond 
with "Foo" and "Bar" on the stack. This might not be possible/hard to do 
since professional Forths inline function calls. 

About Private/Public members, their use in modules (at least in the 
languages I've used) solves two basic problems: 

1) To prevent private members from being used outside the module. 

Most times this is effectively just "advisory". Nothing can stop a 
determined programmer from calling a private function. 

However, when running "external/untrusted" code (ie, in a sandboxed 
environment), the public/private distinction is a simple way to determine 
what code can be called from untrusted code. Ruby on Rails uses this 
simple mechanism very effectively. Java has this too along with a much 
more fine-grained system of access controls. But in both cases the 
sandboxing is very much tied with the OOP and reflective capabilities 
that both languages support "out of the box". Forth lacks both OOP and 
reflection (that is, out of the box), and IMO, it would be a mistake to 
use public/private distinctions for this particular purpose.

Even as a Forth newcomer, I feel that preventing REPL access to private 
members is a very bad idea, since it would be impossible to debug code 
from the commandline. Not everyone relies on just reading the source code 
printout, as is customary (and indeed, the only possible way) in most 
other languages. Other languages rely on editors to follow function 
declarations in debugging. Examples are Eclipse with Java. Having used 
Eclipse/Java regularly, IMHO, this is not as effective as Forth's 
ability to do this from the REPL, and it would be a step backwards if the 
private/public distinction curtailed its use. 

So, in summary, for Forth, the private/public distinction *in this 
context* is not useful. 

For sandboxing, a separate mechanism should be proposed. 

2) The other basic use of public/private distinctions is to enforce a 
client-facing "API", or "interface" for a module. The reason for this is 
to decouple the API/interface from its implementation. 

I feel this is a valid reason for the public/private distinction, but it 
need only be advisory. Also, only "public" members need to be identified. 
What might be useful is an "api?" function that lists the public members 
of a module.

I personally would prefer a "poor man's" alternative -- simply indicate 
public members by a consitent naming convention. No extra plumbing 
needed. The "api?" function could simply parse the names of each member 
based on this naming convention. 

Cheers
Arnold

Arnold Doray <thinksquared@gmail.com> writes:
>On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:
>
>> What should modules achieve.
>
>I feel modules are useful mainly because they help solve the problem of 
>name clashes when writing and using libraries. 
>
>For example: 
>
>module: Foo
>begin-module
>
>: say ." Foo " ;
>
>end-module
>
>module: Bar
>begin-module
>
>: say ." Bar " ;
>
>end-module
>
>\ Used in a named or anonymous module
>begin-module
>Foo \ Depends on Foo
>Bar \ Depends on Bar
>
>: say ." My " Foo.say Bar.say
>
>end-module
>
>OR if there were no ambiguity about what "say" was: 
>
>: say2 ." my " say ;  
>
>(Of course, I'm not insisting on the Java "." syntax.) 
>
>Is this practicable? Are there better alternatives? 
[...]
>I personally would prefer a "poor man's" alternative -- simply indicate 
>public members by a consitent naming convention.

A simple alternative is to indicate private members by a consistent
naming convention:

\ "module" foo
: foo-say ." Foo " ;

\ "module" bar
: bar-say ." Bar " ;

\ top-level
: say ." My " foo-say bar-say ;

> No extra plumbing 
>needed.

Yes.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

On Oct 19, 9:50=A0am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
wrote:
> A simple alternative is to indicate private members by a consistent
> naming convention:

Quite. Except for the dependency and version control, which seem to be
latent capacities of only some of the various public/private systems
under discussion, the modular programming is automating something that
can be done by hand.

That automation of something that would be otherwise done by hand is,
of course, an advantage for a library accepting contributions, where
being able to freely accommodate individual quirks in coding style in
the PRIVATE section increases the likelihood of getting contributions.

On 19.10.2011 15:37, Arnold Doray wrote:
> On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:
>
>> What should modules achieve.
>
> I feel modules are useful mainly because they help solve the problem of
> name clashes when writing and using libraries.

IMO one should clearly differentiate between the concept of modules and 
the concept of libraries.

Modules are source code packages. Whether they hide names or not when 
compiled is secondary.

Libraries are portable code objects with clear API definition. Whether 
they are precompiled or in source format is secondary.

Andreas

On Oct 19, 4:26=A0pm, "A. K." <a...@nospam.org> wrote:
> On 19.10.2011 15:37, Arnold Doray wrote:
>> On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:
>>> What should modules achieve.
>
>> I feel modules are useful mainly because they help solve the
>> problem of name clashes when writing and using libraries.

> IMO one should clearly differentiate between the concept of modules and
> the concept of libraries.

Note, however, that doing so does not undermine the above.

> Modules are source code packages. Whether they hide names or not when
> compiled is secondary.

The question is then in what context is it most useful to package
Forth source code. Without arguing the question in detail, one
plausible response is for libraries of source code.

> Libraries are portable code objects with clear API definition. Whether
> they are precompiled or in source format is secondary.

With libraries of Forth source code, a clear API definition entails
specifying the the behavior of the words that make up the API. However
well factored Forth source will, naturally, contain more words in the
lexicon than specified in the API ~ and, indeed, with provisions to
cope with implementation dependencies on different hosts, may well
vary from compile to compile of the same code source.

For source code libraries that have been packaged into modules, a
system for designating the words that make up the API and those that
do not is fairly natural.

On Oct 19, 8:50=A0am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
wrote:
> Arnold Doray <thinksqua...@gmail.com> writes:
> >On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:
>
> >> What should modules achieve.
>
> >I feel modules are useful mainly because they help solve the problem of
> >name clashes when writing and using libraries.
>
> >For example:
>
> >module: Foo
> >begin-module
>
> >: say ." Foo " ;
>
> >end-module
>
> >module: Bar
> >begin-module
>
> >: say ." Bar " ;
>
> >end-module
>
> >\ Used in a named or anonymous module
> >begin-module
> >Foo \ Depends on Foo
> >Bar \ Depends on Bar
>
> >: say ." My " Foo.say Bar.say
>
> >end-module
>
> >OR if there were no ambiguity about what "say" was:
>
> >: say2 ." my " say ; =A0
>
> >(Of course, I'm not insisting on the Java "." syntax.)
>
> >Is this practicable? Are there better alternatives?
> [...]
> >I personally would prefer a "poor man's" alternative -- simply indicate
> >public members by a consitent naming convention.
>
> A simple alternative is to indicate private members by a consistent
> naming convention:
>
> \ "module" foo
> : foo-say ." Foo " ;
>
> \ "module" bar
> : bar-say ." Bar " ;
>
> \ top-level
> : say ." My " foo-say bar-say ;
>
> > No extra plumbing
> >needed.
>
> Yes.
>
> - anton
> --
> M. Anton Ertl =A0http://www.complang.tuwien.ac.at/anton/home.html
> comp.lang.forth FAQs:http://www.complang.tuwien.ac.at/forth/faq/toc.html
> =A0 =A0 =A0New standard:http://www.forth200x.org/forth200x.html
> =A0 =A0EuroForth 2011:http://www.euroforth.org/ef11/


Haven't the C programmers had this discussion a couple of decades ago?
The argument about prefix naming to avoid name clashes vs name reuse?
And, isn't it one reason why modular programming and OO programming
came into existence?

Krishna

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> 
> Haven't the C programmers had this discussion a couple of decades ago?

Not that I know of: C++ has namespaces, C doesn't.

> The argument about prefix naming to avoid name clashes vs name reuse?
> And, isn't it one reason why modular programming and OO programming
> came into existence?

Modular programming, maybe, but not OO programming, which solves a
different problem.  Many OO languages have modules, packages,
"public", "private", and so on, so it's easy for people to assume
that's a part of OO.  But OO is really about behaviour versus state:
you interact with an object via its methods, not its data.

Andrew.

On Oct 20, 3:57=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > Haven't the C programmers had this discussion a couple of decades ago?
>
> Not that I know of: C++ has namespaces, C doesn't.
>
> > The argument about prefix naming to avoid name clashes vs name reuse?
> > And, isn't it one reason why modular programming and OO programming
> > came into existence?
>
> Modular programming, maybe, but not OO programming, which solves a
> different problem. =A0Many OO languages have modules, packages,
> "public", "private", and so on, so it's easy for people to assume
> that's a part of OO. =A0But OO is really about behaviour versus state:
> you interact with an object via its methods, not its data.
>

In OO programming, data is usually private to an object, while the
methods constitute the public interface. In fact, one of the rules of
thumb given by Stroustrup [1] is:

"Don't use public data members"

Also, from ref. 1, in giving a rationale for going beyond the
traditional programming methods supported by C, such as procedural
programming and modular programming:

"Over the years, the emphasis in the design of programs has shifted
away from the design of procedures, and toward the organization of
data. Among other things, this reflects an increase in program size. A
set of related procedures with the data they manipulate is often
called a *module* ... This paradigm is known as the 'data-hiding
prinicple.' Where there is no grouping of procedures with related
data, the procedural programming style suffices. ..."

Stroustrup then goes on to talk about limitations of the modular
approach, and the need for abstract data types.


Forth appears to be mostly stuck in the procedural programming
paradigm, which limits its scalability.

Krishna
--

[1] B. Stroustrup, The C++ Programming Language, 2nd ed,, AT&T (1993).

Krishna Myneni <krishna.myneni@ccreweb.org> wrote:
> On Oct 20, 3:57?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> wrote:
>> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>>
>> > Haven't the C programmers had this discussion a couple of decades ago?
>>
>> Not that I know of: C++ has namespaces, C doesn't.
>>
>> > The argument about prefix naming to avoid name clashes vs name reuse?
>> > And, isn't it one reason why modular programming and OO programming
>> > came into existence?
>>
>> Modular programming, maybe, but not OO programming, which solves a
>> different problem. Many OO languages have modules, packages,
>> "public", "private", and so on, so it's easy for people to assume
>> that's a part of OO. But OO is really about behaviour versus state:
>> you interact with an object via its methods, not its data.
> 
> In OO programming, data is usually private to an object, while the
> methods constitute the public interface. In fact, one of the rules of
> thumb given by Stroustrup [1] is:
> 
> "Don't use public data members"

Indeed.  That's exactly what I said, I think.

Andrew.

On Oct 20, 6:55=A0am, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> > On Oct 20, 3:57?am, Andrew Haley <andre...@littlepinkcloud.invalid>
> > wrote:
> >> Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> >> > Haven't the C programmers had this discussion a couple of decades ag=
o?
>
> >> Not that I know of: C++ has namespaces, C doesn't.
>
> >> > The argument about prefix naming to avoid name clashes vs name reuse=
?
> >> > And, isn't it one reason why modular programming and OO programming
> >> > came into existence?
>
> >> Modular programming, maybe, but not OO programming, which solves a
> >> different problem. Many OO languages have modules, packages,
> >> "public", "private", and so on, so it's easy for people to assume
> >> that's a part of OO. But OO is really about behaviour versus state:
> >> you interact with an object via its methods, not its data.
>
> > In OO programming, data is usually private to an object, while the
> > methods constitute the public interface. In fact, one of the rules of
> > thumb given by Stroustrup [1] is:
>
> > "Don't use public data members"
>
> Indeed. =A0That's exactly what I said, I think.
>
> Andrew.

We may be talking past each other a bit. Actually, I will take back
one thing which I said concerning Forth being stuck in the procedural
programming paradigm. The concepts of data hiding and name reuse are
distinct in most languages, but are intimately related in Forth. Forth
supports a form of modular programming by allowing data hiding for a
set of procedures (words) through name reuse. So, if we have a set of
data, defined as constants, variables, arrays, etc., named A B C ...
etc. This data may be hidden, after the procedures are defined, by
redefining A B C ... The instance of data for the first module then
becomes hidden, and the new set of procedures can use the recent
definitions of A B C ... as its hidden data.

In the modular programming concept proposed here, we aim for a more
formal way of implementing modular programming.

Krishna

Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>On Oct 19, 8:50=A0am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
>wrote:
>Haven't the C programmers had this discussion a couple of decades ago?

Who knows?  In any case, it has not had an effect on the language, and
that's despite the fact that name clashes are a much more severe
problem in C than in Forth: If there is a name clash in C, the program
won't build.  If there is a name clash in Forth, the program will
usually compile and run as intended.

>The argument about prefix naming to avoid name clashes vs name reuse?
>And, isn't it one reason why modular programming and OO programming
>came into existence?

The above-mentioned property of Algol-like languages like C was
certainly a contributing factor.  A culture of software development
that makes it hard to resolve name clashes by renaming is probably
another factor.  I am a fan of accomodating various software
development cultures, but since Forth has nicer properties, and name
hiding is in conflict with the culture of interactive testing and
debugging, name hiding is questionable in Forth.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

On Wed, 19 Oct 2011 13:50:35 +0000, Anton Ertl wrote:

> Arnold Doray <thinksquared@gmail.com> writes:
>>On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:
>>
>>> What should modules achieve.
>>
>>I feel modules are useful mainly because they help solve the problem of
>>name clashes when writing and using libraries.
>>
>>For example:
>>
>>module: Foo
>>begin-module
>>
>>: say ." Foo " ;
>>
>>end-module
>>
>>module: Bar
>>begin-module
>>
>>: say ." Bar " ;
>>
>>end-module
>>
>>\ Used in a named or anonymous module begin-module
>>Foo \ Depends on Foo
>>Bar \ Depends on Bar
>>
>>: say ." My " Foo.say Bar.say
>>
>>end-module
>>
>>OR if there were no ambiguity about what "say" was:
>>
>>: say2 ." my " say ;
>>
>>(Of course, I'm not insisting on the Java "." syntax.)
>>
>>Is this practicable? Are there better alternatives?
> [...]
>>I personally would prefer a "poor man's" alternative -- simply indicate
>>public members by a consitent naming convention.
> 
> A simple alternative is to indicate private members by a consistent
> naming convention:
> 
> \ "module" foo
> : foo-say ." Foo " ;
> 
> \ "module" bar
> : bar-say ." Bar " ;
> 
> \ top-level
> : say ." My " foo-say bar-say ;
> 
>> No extra plumbing
>>needed.
> 
> Yes.
> 
> - anton

Not what I meant. I meant a naming convention to distinguish private vs 
public members. 

Your solution is for a different problem - resolving name clashes. 

While the solution you propose is indeed simple, it has some serious 
drawbacks. 

1) It does not promote separation of the API/interface from the 
implementation. 

For example, if I decided to upgrade from module "Foo" to module "Bar" 
then I'd have to recode (or get my editor to refactor) all calls from  
foo-xxx to bar-xxx. Ugly no? 

2) Also, what happens if you have a module named "foo-bar" and module 
"foo" with functions "say" and "bar-say" respectively? So, you'd have to 
resort to more elaborate hand-prefixing conventions. Where does it end? I 
would prefer my module system to handle that sort of plumbing issues.  
 
3) I feel having to prefix in every function and constant to gain access 
to every module I use and write a deal-breaker. 

So, no, I don't think this simple solution is a practical one. 

Cheers,
Arnold






Cheers,
Arnold

On Wed, 19 Oct 2011 13:50:35 +0000, Anton Ertl wrote:

> Arnold Doray <thinksquared@gmail.com> writes:
>>On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:
>>
>>> What should modules achieve.
>>
>>I feel modules are useful mainly because they help solve the problem of
>>name clashes when writing and using libraries.
>>
>>For example:
>>
>>module: Foo
>>begin-module
>>
>>: say ." Foo " ;
>>
>>end-module
>>
>>module: Bar
>>begin-module
>>
>>: say ." Bar " ;
>>
>>end-module
>>
>>\ Used in a named or anonymous module begin-module
>>Foo \ Depends on Foo
>>Bar \ Depends on Bar
>>
>>: say ." My " Foo.say Bar.say
>>
>>end-module
>>
>>OR if there were no ambiguity about what "say" was:
>>
>>: say2 ." my " say ;
>>
>>(Of course, I'm not insisting on the Java "." syntax.)
>>
>>Is this practicable? Are there better alternatives?
> [...]
>>I personally would prefer a "poor man's" alternative -- simply indicate
>>public members by a consitent naming convention.
> 
> A simple alternative is to indicate private members by a consistent
> naming convention:
> 
> \ "module" foo
> : foo-say ." Foo " ;
> 
> \ "module" bar
> : bar-say ." Bar " ;
> 
> \ top-level
> : say ." My " foo-say bar-say ;
> 
>> No extra plumbing
>>needed.
> 
> Yes.
> 
> - anton

Not what I meant. I meant a naming convention to distinguish private vs 
public members. 

Your solution is for a different problem - resolving name clashes. 

While the solution you propose is indeed simple, it has some serious 
drawbacks. 

1) It does not promote separation of the API/interface from the 
implementation. 

For example, if I decided to upgrade from module "Foo" to module "Bar" 
then I'd have to recode (or get my editor to refactor) all calls from  
foo-xxx to bar-xxx. Ugly no? 

2) Also, what happens if you have a module named "foo-bar" and module 
"foo" with functions "say" and "bar-say" respectively? So, you'd have to 
resort to more elaborate hand-prefixing conventions. Where does it end? I 
would prefer my module system to handle that sort of plumbing issues.  
 
3) I feel having to prefix in every function and constant to gain access 
to every module I use and write a deal-breaker. 

So, no, I don't think this simple solution is a practical one. 

Cheers,
Arnold






Cheers,
Arnold

On Thu, 20 Oct 2011 13:19:46 +0000, Anton Ertl wrote:

> Who knows?  In any case, it has not had an effect on the language, and
> that's despite the fact that name clashes are a much more severe problem
> in C than in Forth: If there is a name clash in C, the program won't
> build.  If there is a name clash in Forth, the program will usually
> compile and run as intended.

At the risk of venturing into some serious speculation, I'd say it has 
limited Forth's use in writing *larger* systems. 

When I read the old JFAR articles, John V. Noble's tutorials and Brodie's 
books, they seem to have envisioned Forth as a general purpose, flexible 
language, suitable for building large applications.

I image they would have been very surprised to learn (then) that Forth in 
the 21st century is stuck in its present usage paradigm -- embedded 
programming. 

I would argue that especially for Forth, successful "application" 
programming requires solid module support to solve the name clash 
problem. 

You can't compare "C" and Forth on this issue. 

"C" does not encourage factoring code to the extent that Forth does. So 
name clashes are a serious, practical issue in Forth, especially if you 
are writing libraries. 

And for application programming to thrive, you need libraries. 

Cheers,
Arnold

On Wed, 19 Oct 2011 22:26:42 +0200, A. K. wrote:

> On 19.10.2011 15:37, Arnold Doray wrote:
>> On Sun, 09 Oct 2011 17:29:56 -0700, jacko wrote:
>>
>>> What should modules achieve.
>>
>> I feel modules are useful mainly because they help solve the problem of
>> name clashes when writing and using libraries.
> 
> IMO one should clearly differentiate between the concept of modules and
> the concept of libraries.
> 
> Modules are source code packages. Whether they hide names or not when
> compiled is secondary.
> 
> Libraries are portable code objects with clear API definition. Whether
> they are precompiled or in source format is secondary.
> 
> Andreas

I would be hard pressed to improve on Bruce's response to your valid 
observations. 

May I add that I feel that any Forth module system that does not cleanly 
solve the name clash problem from both source code and REPL perspectives 
is doomed to fail, because it brings little value in writing larger 
systems. 

Chuck Moore has commented on this issue -- he hasn't found a solution 
he's happy with. (unfortunately, I can't remember the source). So a Forth 
module system that solves this practical issue is a very worth goal, 
IMHO. 

Cheers,
Arnold

 

On 10/20/11 10:32 AM, Arnold Doray wrote:
> On Thu, 20 Oct 2011 13:19:46 +0000, Anton Ertl wrote:
>
>> Who knows?  In any case, it has not had an effect on the language, and
>> that's despite the fact that name clashes are a much more severe problem
>> in C than in Forth: If there is a name clash in C, the program won't
>> build.  If there is a name clash in Forth, the program will usually
>> compile and run as intended.
>
> At the risk of venturing into some serious speculation, I'd say it has
> limited Forth's use in writing *larger* systems.
>
> When I read the old JFAR articles, John V. Noble's tutorials and Brodie's
> books, they seem to have envisioned Forth as a general purpose, flexible
> language, suitable for building large applications.
>
> I image they would have been very surprised to learn (then) that Forth in
> the 21st century is stuck in its present usage paradigm -- embedded
> programming.

There are worse markets to be "stuck" in. Some very large percentage of 
processors are used in embedded systems.  Forth was designed for 
embedded systems, and to say that this is still a major area for Forth 
is not a complaint, it's an acknowledgement that it's been successful in 
the market it was designed for.

> I would argue that especially for Forth, successful "application"
> programming requires solid module support to solve the name clash
> problem.

Having been involved is several large projects (involving networks of 
hundreds of embedded systems and a few larger ones), I can honestly say 
that name clashes have never been a serious issue.

> You can't compare "C" and Forth on this issue.
>
> "C" does not encourage factoring code to the extent that Forth does. So
> name clashes are a serious, practical issue in Forth, especially if you
> are writing libraries.
>
> And for application programming to thrive, you need libraries.

That depends on the kinds of applications you're writing. It is more 
true in PC systems than in embedded systems (where stuff tends to be 
very custom and specialized), but in PC systems there are good 
mechanisms for managing libraries.  I have never had a problem with name 
clashes there, either.

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Thu, 20 Oct 2011 14:25:21 -1000, Elizabeth D. Rather wrote:

>>
>> I image they would have been very surprised to learn (then) that Forth
>> in the 21st century is stuck in its present usage paradigm -- embedded
>> programming.
> 
> There are worse markets to be "stuck" in. Some very large percentage of
> processors are used in embedded systems.  Forth was designed for
> embedded systems, and to say that this is still a major area for Forth
> is not a complaint, it's an acknowledgement that it's been successful in
> the market it was designed for.
> 

"Stuck" was a poor word choice. In no way was I disparaging embedded 
programming. I was merely pointing out what appears (to me) to be a huge 
disconnect between the expectations of the past and the reality of the 
present. 

Yes, Forth was designed originally for "embedded" programming. However, 
other languages also had other sweetspots originally - Lisp, Java, C, 
Ruby and ML come to mind - but have grown beyond them. I would be unwise 
to insist that Lisp in 2011 is only good for AI or Java for applets. 

>> I would argue that especially for Forth, successful "application"
>> programming requires solid module support to solve the name clash
>> problem.
> 
> Having been involved is several large projects (involving networks of
> hundreds of embedded systems and a few larger ones), I can honestly say
> that name clashes have never been a serious issue.
> 

The question is how large was each individual system? A large collection 
of simple systems is not a "large" system, as I intend it to mean here. 
For example, 1000's of handheld Forth barcode scanners communicating with 
a central server(s) is not a "large" system. 

A large system might be an application server backed by a database, with 
support for HTTPS, and a macro language to generate HTML (eg, like JSP) 
and generate PDF. 

>> You can't compare "C" and Forth on this issue.
>>
>> "C" does not encourage factoring code to the extent that Forth does. So
>> name clashes are a serious, practical issue in Forth, especially if you
>> are writing libraries.
>>
>> And for application programming to thrive, you need libraries.
> 
> That depends on the kinds of applications you're writing. It is more
> true in PC systems than in embedded systems (where stuff tends to be

Yes, that's exactly my point. Embedded systems are very customized and 
eschew the use of modules whenever possible, for a variety of reasons. 
Successful embedded programming is radically different from successful 
"application" programming. Easy application programming requires module 
support.

> very custom and specialized), but in PC systems there are good
> mechanisms for managing libraries.  I have never had a problem with name
> clashes there, either.

Could you elaborate on these good mechanisms? Are they portable (ANS 
Forth)? 

Here's one area I think Forth could really shine -- the same space where 
Java now is mainly used -- server side programs. These are very much like 
embedded programs, but you need good module support. 

Cheers,
Arnold

On Fri, 21 Oct 2011 12:26:12 +0000 (UTC), Arnold Doray
<thinksquared@gmail.com> wrote:

>The question is how large was each individual system? A large collection 
>of simple systems is not a "large" system, as I intend it to mean here. 
>For example, 1000's of handheld Forth barcode scanners communicating with 
>a central server(s) is not a "large" system. 

Does a commercial PC app with over 1,000,000 lines of Forth source
code count as large?

Stephen


-- 
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

On Oct 21, 8:38=A0am, stephen...@mpeforth.com (Stephen Pelc) wrote:
> Does a commercial PC app with over 1,000,000 lines of
> Forth source code count as large?

You often bring up this example, but the existence of a handful of
large (in terms of source code) Forth applications doesn't in and of
itself say much of anything.  And if it says anything, it says that
because you have to repeatedly point to the same handful of large
Forth applications that Forth doesn't effectively compete in that
space.

It takes zero imagination to see it's entirely possible for far less
capable and far more cumbersome programming languages being used to
create million-line applications.  It should surprise absolutely
nobody that somewhere in the world, there is always going to be at
least one example of some large application written in pretty much
every programming language created.  Big deal, so what.

So let's dive into the details that are probably more relevant than
source lines of code to this discussion of modules:

1.  How many programmers worked on the application you cite?

2.  How did these programmers work?  For example, were specific people
assigned specific sections of code?  Were there detailed design
documents up front that and everyone just chugged through the
implementation?

3.  What rules (if any) were set up for developers to follow to avoid
things like name clashes and other "programming in the large" issues?

4.  What kinds of problems were found with the process during
developement?  What kind of solutions were invented to solve them?

5.  Was any of the code factored out for reuse in other projects
(maybe not as a formal "module" but in some form)?

6.  How long did the development last?

The point of these questions (and probably dozens of others I could
come up with) is to understand not if it is merely possible to write
million-line Forth applications.  Honestly, the only people who find
that interesting are people who don't matter.  The point is to drive
out a better understanding of where Forth worked well *and* where
Forth could be improved.

Or put another way, while you can point to a handful of large Forth
applications that have been created over the years, there are orders
of magnitude more examples of large applications being written in
other languages.  Why?  It isn't just tenacity.  It's because those
languages (and the cultures associated with those languages) have
worked out tools, techniques, and processes for being able to
confidently push forward on developing such large applications.

And confidence matters:  If you asked me to create a million line
program in something like Java or C# or C++, I would would be able to
draw on the experience of many people over many years.  I would be
able to research best practices for managing such a project.  I would
be able to point to commercial and open-source tools that are helpful
in managing the project, testing the quality of code being written,
testing how well the software tracked requirements, etc.  With Forth,
I have none of that.  And when people are considering implementation
languages look at Forth for large-scale projects, *that* is what they
are looking for.  Not just you saying that you can point to an
existing million-line application and say "we did that in Forth".

Personally, when someone says a project is large, I don't care much
about the number of lines of source.  Well, I care, but mostly because
it probably relates more to ergonomic issues (like if the developers
are likely to have carpal tunnel syndrome from all that typing).  What
is far more interesting when people talk about "large" are related to
things like managing the work of a large number of developers or
tracking the progress on a large number of requirements or testing the
quality of a large number of definitions.  Talk to me more about that
sense of "large", please, and talk more about how Forth helps or
hinders that.

On 10/21/11 2:26 AM, Arnold Doray wrote:
> On Thu, 20 Oct 2011 14:25:21 -1000, Elizabeth D. Rather wrote:
>
>>>
>>> I image they would have been very surprised to learn (then) that Forth
>>> in the 21st century is stuck in its present usage paradigm -- embedded
>>> programming.
>>
>> There are worse markets to be "stuck" in. Some very large percentage of
>> processors are used in embedded systems.  Forth was designed for
>> embedded systems, and to say that this is still a major area for Forth
>> is not a complaint, it's an acknowledgement that it's been successful in
>> the market it was designed for.
>>
>
> "Stuck" was a poor word choice. In no way was I disparaging embedded
> programming. I was merely pointing out what appears (to me) to be a huge
> disconnect between the expectations of the past and the reality of the
> present.
>
> Yes, Forth was designed originally for "embedded" programming. However,
> other languages also had other sweetspots originally - Lisp, Java, C,
> Ruby and ML come to mind - but have grown beyond them. I would be unwise
> to insist that Lisp in 2011 is only good for AI or Java for applets.

Languages flourish or not for reasons that go far beyond their technical 
capabilities.  Java, for example, has been aggressively promoted by Sun, 
with more global marketing power than you can imagine. In the late 90's 
FORTH, Inc. developed a "smart card" programming system that we 
benchmarked against the [then new] Java Card: it way outperformed Java 
Card in every dimension that card suppliers care about (size of object, 
speed, programming flexibility, security), but we didn't have a 
thousandth of the marketing budget that Sun had, and we went virtually 
unnoticed.  I gave a presentation once in the headquarters of VISA, at 
the end of which the head of smart card programming said that she agreed 
our system was technically superior, but "the industry is going to adopt 
Java Card, so we need to as well."

>>> I would argue that especially for Forth, successful "application"
>>> programming requires solid module support to solve the name clash
>>> problem.
>>
>> Having been involved is several large projects (involving networks of
>> hundreds of embedded systems and a few larger ones), I can honestly say
>> that name clashes have never been a serious issue.
>
> The question is how large was each individual system? A large collection
> of simple systems is not a "large" system, as I intend it to mean here.
> For example, 1000's of handheld Forth barcode scanners communicating with
> a central server(s) is not a "large" system.

In the 1980's a company had a failed project on which 35 programmers had 
worked for 3 years but failed to deliver adequate performance or provide 
significant functionality.  We were brought in and in 6 mos. has 
equivalent functionality to the prior group and 10x better performance. 
In 18 mos. we successfully completed the project. Our total team was 15 
programmers, of which about half had not seen Forth prior to the 
project. It featured a central database on 4 shared servers, with ~500 
fairly complex remotes doing different functionality.

....

>
>> very custom and specialized), but in PC systems there are good
>> mechanisms for managing libraries.  I have never had a problem with name
>> clashes there, either.
>
> Could you elaborate on these good mechanisms? Are they portable (ANS
> Forth)?

Well, take DLLs for example.  That's the standard mechanism for 
libraries on PCs. Fprths can generate DLLs which, once generated, are 
indistinguishable from DLLs generated by anythin else.

> Here's one area I think Forth could really shine -- the same space where
> Java now is mainly used -- server side programs. These are very much like
> embedded programs, but you need good module support.

Java is probably unbeatable in that arena for political reasons 
(well-known, highly-regarded, popular, etc.), regardless of technical 
issues. Forth could not compete without extensive marketing support, 
which isn't likely to happen.

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Fri, 21 Oct 2011 12:38:47 +0000, Stephen Pelc wrote:

> On Fri, 21 Oct 2011 12:26:12 +0000 (UTC), Arnold Doray
> <thinksquared@gmail.com> wrote:
> 
>>The question is how large was each individual system? A large collection
>>of simple systems is not a "large" system, as I intend it to mean here.
>>For example, 1000's of handheld Forth barcode scanners communicating
>>with a central server(s) is not a "large" system.
> 
> Does a commercial PC app with over 1,000,000 lines of Forth source code
> count as large?
> 
> Stephen

That is a pretty impressive feat. 

Given say 10 lines per definition, you must have used at least 100,000 
words! 

What strategies did your team adopt to avoid name clashes? Was a module 
system used to manage the complexity? Do elaborate. 

I am sure your insights would be beneficial to other Forth newcomers on 
clf like myself. 

Cheers,
Arnold

Arnold Doray <thinksquared@gmail.com> writes:
>On Thu, 20 Oct 2011 13:19:46 +0000, Anton Ertl wrote:
>
>> Who knows?  In any case, it has not had an effect on the language, and
>> that's despite the fact that name clashes are a much more severe problem
>> in C than in Forth: If there is a name clash in C, the program won't
>> build.  If there is a name clash in Forth, the program will usually
>> compile and run as intended.
>
>At the risk of venturing into some serious speculation, I'd say it has 
>limited Forth's use in writing *larger* systems. 

I doubt it.  The lack of a module system has not limited C's use in
writing larger systems, like Netscape, the Linux kernel, Emacs, gcc.

>I would argue that especially for Forth, successful "application" 
>programming requires solid module support to solve the name clash 
>problem. 
>
>You can't compare "C" and Forth on this issue. 
>
>"C" does not encourage factoring code to the extent that Forth does.

Ok.  So a program with 10,000 names is larger in idiomatic C than in
idiomatic Forth.  Let's see:

[~/gforth/engine:76225] nm -A --defined-only dblsub.o io.o signals.o main-itc-noll.o |wc -l
136
[~/gforth/engine:76226] wc -l dblsub.c io.c signals.c main.c
    65 dblsub.c
   764 io.c
   462 signals.c
  2442 main.c
  3733 total

So we have about 27 lines per global symbol.

Now let's look at some Forth code:

[~/gforth:76257] gforth -e "hashpop ? cr bye"
2983 
[~/gforth:76258] gforth -i kernl64l.fi -e "words bye"|tr ' ' '\n'|tr -s '\n'|wc -l
1016

So, 2983-1016=1967 names outside the kernel.

[~/gforth:76260] wc -l `gforth -e ".included bye"` -e ".included bye"
  ...
  9090 total

So, about 4.6 lines per name.

So if we are able to write 10M line systems in C (not sure how big the
Linux kernel is at the moment), we should at least be able to write
1.7M line applications in Forth.

And that's if name clashes would cause as big a problem in Forth as in
C, which they don't.

Of course, the idea of writing such large systems is considered
horrible by a significant part of the Forth community.

> So 
>name clashes are a serious, practical issue in Forth, especially if you 
>are writing libraries. 
>
>And for application programming to thrive, you need libraries. 

I think there are much larger impediments to the use of libraries and
to writing large systems in Forth than name clash problems.  In
particular, Forth already has a standard way to hide words (with
wordlists and search order manipulation) if that is really beneficial
in the overall balance.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

On Fri, 21 Oct 2011 21:29:45 +0000 (UTC), Arnold Doray
<thinksquared@gmail.com> wrote:

>On Fri, 21 Oct 2011 12:38:47 +0000, Stephen Pelc wrote:
>> Does a commercial PC app with over 1,000,000 lines of Forth source code
>> count as large?

>Given say 10 lines per definition, you must have used at least 100,000 
>words! 

103,084 words including the VFX Forth system, 7287 words.

This is not an MPE app. It's an app by a client. The app has evolved
over 25 years and a number of CPUs and Forth systems. Because it is
a client's app, I am limited in what I can say.

The basic team approach is the surgical team approach from Fred
Brooks' 'Mythical Man Month'. There are about half a dozen primary
developers, plus testers.

>What strategies did your team adopt to avoid name clashes? Was a module 
>system used to manage the complexity? Do elaborate.

The toolmakers tend to use the module system from VFX Forth. This just
allows named modules with words exported to the surrounding wordlist
or vocabulary.

Application programmers use the internal module mechanism which just
uses two additional wordlists.

Name clashes are mostly avoided by naming conventions, discipline and 
ruthless removal of name conflicts.

IMHO the success of large applications is mostly due to project
management. Where Forth scores is in ease of testing.

Stephen


-- 
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

On Oct 22, 11:19=A0am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
wrote:

> I think there are much larger impediments to the use of libraries and
> to writing large systems in Forth than name clash problems. =A0In
> particular, Forth already has a standard way to hide words (with
> wordlists and search order manipulation) if that is really beneficial
> in the overall balance.

The modular programming systems under discussion are all essentially
means to automate specific routine wordlist and search order
manipulations. One appeal of automating these manipulations in a
library accepting contributions is consistency across the library.

After experiment, I prefer a modular programming lexicon for libraries
that is open to a range of implementation approaches, so the caller
can pick their poison.

In article <2011Oct22.171943@mips.complang.tuwien.ac.at>,
Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:
>Arnold Doray <thinksquared@gmail.com> writes:
>>On Thu, 20 Oct 2011 13:19:46 +0000, Anton Ertl wrote:
>>
>>> Who knows?  In any case, it has not had an effect on the language, and
>>> that's despite the fact that name clashes are a much more severe problem
>>> in C than in Forth: If there is a name clash in C, the program won't
>>> build.  If there is a name clash in Forth, the program will usually
>>> compile and run as intended.
>>
>>At the risk of venturing into some serious speculation, I'd say it has
>>limited Forth's use in writing *larger* systems.
>
>I doubt it.  The lack of a module system has not limited C's use in
>writing larger systems, like Netscape, the Linux kernel, Emacs, gcc.

Unix has a module system in place. It is called executables. They
communicate through files, pipes and sockets. They made it possible
to make a large system on a 16 bit computer like PDP11.

Groetjes Albert

--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

On Oct 22, 11:18=A0am, stephen...@mpeforth.com (Stephen Pelc) wrote:
> On Fri, 21 Oct 2011 21:29:45 +0000 (UTC), Arnold Doray
>
> <thinksqua...@gmail.com> wrote:
> >On Fri, 21 Oct 2011 12:38:47 +0000, Stephen Pelc wrote:
> >> Does a commercial PC app with over 1,000,000 lines of Forth source cod=
e
> >> count as large?
> >Given say 10 lines per definition, you must have used at least 100,000
> >words!
>
> 103,084 words including the VFX Forth system, 7287 words.
>
> This is not an MPE app. It's an app by a client. The app has evolved
> over 25 years and a number of CPUs and Forth systems. Because it is
> a client's app, I am limited in what I can say.
>
> The basic team approach is the surgical team approach from Fred
> Brooks' 'Mythical Man Month'. There are about half a dozen primary
> developers, plus testers.
>
> >What strategies did your team adopt to avoid name clashes? Was a module
> >system used to manage the complexity? Do elaborate.
>
> The toolmakers tend to use the module system from VFX Forth. This just
> allows named modules with words exported to the surrounding wordlist
> or vocabulary.
>
> Application programmers use the internal module mechanism which just
> uses two additional wordlists.
>
> Name clashes are mostly avoided by naming conventions, discipline and
> ruthless removal of name conflicts.
>

The strategies of code development which you and Elizabeth describe
appear to be specific to a business model of software development.
But, there is another world in which code is written outside of that
model. When developing a library of reusable code in a highly
decentralized and delocalized setting, in which there is no central
authority to impose coding restrictions, it makes sense to allow each
contributor to use whatever naming convention he/she feels is most
appropriate to their work.

There also seems to be a mis-perception that modular code is all about
avoiding name clashes. That's one important facet, but, in addition to
that there are at least two other features which enhance the use of
modular code in applications: a clear API (public interface) that is
obvious from the source itself, and a means by which the module can
assert its own dependencies.

Krishna

On Sun, 23 Oct 2011 05:48:30 -0700 (PDT), Krishna Myneni
<krishna.myneni@ccreweb.org> wrote:

>The strategies of code development which you and Elizabeth describe
>appear to be specific to a business model of software development.

It's not about a business model, it's a managed model.

>But, there is another world in which code is written outside of that
>model. When developing a library of reusable code in a highly
>decentralized and delocalized setting, in which there is no central
>authority to impose coding restrictions, it makes sense to allow each
>contributor to use whatever naming convention he/she feels is most
>appropriate to their work.

It then becomes even more necessary to have some management!
Otherwise, users pay penalties. You choice is a bit of pain
up front, or a lot of pain distributed in small chunks.

>There also seems to be a mis-perception that modular code is all about
>avoiding name clashes. That's one important facet, but, in addition to
>that there are at least two other features which enhance the use of
>modular code in applications: a clear API (public interface) that is
>obvious from the source itself, and a means by which the module can
>assert its own dependencies.

Sure, but I (for one) have always been a fan of good documentation.

Stephen


-- 
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

On Oct 23, 8:54=A0am, stephen...@mpeforth.com (Stephen Pelc) wrote:
> On Sun, 23 Oct 2011 05:48:30 -0700 (PDT), Krishna Myneni
>
> <krishna.myn...@ccreweb.org> wrote:
> >The strategies of code development which you and Elizabeth describe
> >appear to be specific to a business model of software development.
>
> It's not about a business model, it's a managed model.
>

It's a question of how rigid a management model to use. Personally, I
think the FSL is a good example of a loosely managed model which has
resulted in a useful library.

> >But, there is another world in which code is written outside of that
> >model. When developing a library of reusable code in a highly
> >decentralized and delocalized setting, in which there is no central
> >authority to impose coding restrictions, it makes sense to allow each
> >contributor to use whatever naming convention he/she feels is most
> >appropriate to their work.
>
> It then becomes even more necessary to have some management!
> Otherwise, users pay penalties. You choice is a bit of pain
> up front, or a lot of pain distributed in small chunks.
>

It's a tradeoff between the pain of using a specific modules system vs
resolving conflicts after the code has been written. The goal is to
permit all library modules to play together in whatever combination is
needed for an application. Consider that any time a new file is added
to the library, without a modules system, there's a potential for
breaking the user interface of any of the existing files. This is just
unwieldy to coordinate in a large distributed project. I would much
rather accept the minimal pain of using a modules system than
contending with the possibility of having to require authors to recode
their contributions every time new code is added.

> >There also seems to be a mis-perception that modular code is all about
> >avoiding name clashes. That's one important facet, but, in addition to
> >that there are at least two other features which enhance the use of
> >modular code in applications: a clear API (public interface) that is
> >obvious from the source itself, and a means by which the module can
> >assert its own dependencies.
>
> Sure, but I (for one) have always been a fan of good documentation.
>

No argument from me about the necessity of good documentation. Self-
documenting interfaces, as is the code with the PUBLIC: and PRIVATE:
mechanism is a bonus.

Krishna

On Oct 23, 10:26=A0am, Krishna Myneni <krishna.myn...@ccreweb.org>
wrote:

> It's a tradeoff between the pain of using a specific modules system vs
> resolving conflicts after the code has been written. The goal is to
> permit all library modules to play together in whatever combination is
> needed for an application. Consider that any time a new file is added
> to the library, without a modules system, there's a potential for
> breaking the user interface of any of the existing files. This is just
> unwieldy to coordinate in a large distributed project. I would much
> rather accept the minimal pain of using a modules system than
> contending with the possibility of having to require authors to recode
> their contributions every time new code is added.

However, avoiding conflict with the public API of existing modules is
a much less daunting task. A master list of the words forming the
public API of each module, and a contributor can simply do a search
for the words they propose to use to see if there are any conflicts.

On Sat, 22 Oct 2011 16:18:53 +0000, Stephen Pelc wrote:

> 
>>What strategies did your team adopt to avoid name clashes? Was a module
>>system used to manage the complexity? Do elaborate.
> 
> The toolmakers tend to use the module system from VFX Forth. This just
> allows named modules with words exported to the surrounding wordlist or
> vocabulary.
> 
> Application programmers use the internal module mechanism which just
> uses two additional wordlists.
> 
> Name clashes are mostly avoided by naming conventions, discipline and
> ruthless removal of name conflicts.
> 
> IMHO the success of large applications is mostly due to project
> management. Where Forth scores is in ease of testing.
> 
> Stephen

Many thanks for this very insightful reply.

To summarize, MPE had to use a number of strategies to solve the name 
clash issue with 10^5 names (and growing, I assume): 

1) VFX Forth's own module system
2) The app's internal module system
3) Naming conventions
4) Manual name clash resolution
5) Wordlists
6) Documentation/Culture ("discipline" ?)
7) Active management oversight

To a Forth newcomer who has had experience with other languages, this 
seems pretty daunting. 

Compare the situation in Java. There are a vast array of high quality 
open source libraries available for it. As both an app developer and 
library writer, I have never *ever* had to worry about name clashes. 

Never ever. 

Say I wanted to dynamically read scientific data from a NetCDF file, run 
some processing and print it to PDF and make it available over a web 
server. I would just get the NetCDF library from UCAR, the PDF library 
from iText, the app server from Tomcat, write the servlet and I would 
NEVER ONCE have to deal with name clashes. 

Is this advantage so hard to appreciate? 

In the Forth world, management and developers would have to expend 
valuable effort managing this issue in addition to productive work like 
delivering functionality and eradicating bugs.  

When developing larger applications, this surely negates some of the 
advantages that Forth brings to the table. 

It's true that this situation rarely arises in embedded development. 
Embedded development shops are necessarily tightly managed. The reliance 
on libraries is eschewed unless perhaps it's something done in-house or 
something you're saddled with by the client. 

But for app development, a solid module system is paramount for success. 

Cheers,
Arnold

On Sun, 23 Oct 2011 05:48:30 -0700, Krishna Myneni wrote:

> There also seems to be a mis-perception that modular code is all about
> avoiding name clashes. That's one important facet, but, in addition to
> that there are at least two other features which enhance the use of
> modular code in applications: a clear API (public interface) that is
> obvious from the source itself, and a means by which the module can
> assert its own dependencies.

Yes, resolving name clashes aren't everything. 

However, IMHO, name clashes are hardest to solve. Additionally, for Forth 
you need to also come up with a solution that feels "natural" from the 
REPL. All module systems I've seen are source-centric, since these other 
languages just don't offer the level of interactivity that Forth has. 

For me, a solution that feels "natural" would:

a) Make introspective functions like see/locate as easy for members in 
the API as for those not in it. 

b) Similarly, make it easy to restrict these introspective functions to a 
given module. 

c) Would not force the module user to be specific about a name (ie, 
specifying which module the name resides) if there was no ambiguity about 
the name. 

I suspect once you've got a clean solution to name clashes, the other two 
issues that are also important (interface/implementation separation and 
module dependencies) would come out easily. Both depend to a large extent 
on a clean solution to name clashes. 

Cheers,
Arnold

On Sun, 23 Oct 2011 08:57:52 -0700, BruceMcF wrote:

> However, avoiding conflict with the public API of existing modules is a
> much less daunting task. A master list of the words forming the public
> API of each module, and a contributor can simply do a search for the
> words they propose to use to see if there are any conflicts.

How would you coordinate public names for modules that are being 
developed independently? 

If you don't, then wouldn't the app developer have to bear the burden of 
resolving name conflicts of the libraries he uses (either through 
renaming, or changing the search order)? 

Cheers,
Arnold 

On 10/23/11 10:28 PM, Arnold Doray wrote:
> On Sat, 22 Oct 2011 16:18:53 +0000, Stephen Pelc wrote:
>
>>
>>> What strategies did your team adopt to avoid name clashes? Was a module
>>> system used to manage the complexity? Do elaborate.
>>
>> The toolmakers tend to use the module system from VFX Forth. This just
>> allows named modules with words exported to the surrounding wordlist or
>> vocabulary.
>>
>> Application programmers use the internal module mechanism which just
>> uses two additional wordlists.
>>
>> Name clashes are mostly avoided by naming conventions, discipline and
>> ruthless removal of name conflicts.
>>
>> IMHO the success of large applications is mostly due to project
>> management. Where Forth scores is in ease of testing.
>>
>> Stephen
>
> Many thanks for this very insightful reply.
>
> To summarize, MPE had to use a number of strategies to solve the name
> clash issue with 10^5 names (and growing, I assume):
>
> 1) VFX Forth's own module system
> 2) The app's internal module system
> 3) Naming conventions
> 4) Manual name clash resolution
> 5) Wordlists
> 6) Documentation/Culture ("discipline" ?)
> 7) Active management oversight
>
> To a Forth newcomer who has had experience with other languages, this
> seems pretty daunting.

That's probably because you're vastly overestimating the "problem" of 
name clashes.  Truly, of all the things I and our customers have had to 
worry about, name clashes are so low on the list I wouldn't have listed 
it at all.  The things that are high on the list include things like 
clarity of requirements, adequacy of test cases, access to custom 
hardware, documentation of custom hardware, reliability of custom 
hardware, and other systematic issues.

And, incidentally, MPE didn't write this application, their customer did.

> Compare the situation in Java. There are a vast array of high quality
> open source libraries available for it. As both an app developer and
> library writer, I have never *ever* had to worry about name clashes.
>
> Never ever.

Well, I don't worry about them, either, so I'm not sure what that proves.

> Say I wanted to dynamically read scientific data from a NetCDF file, run
> some processing and print it to PDF and make it available over a web
> server. I would just get the NetCDF library from UCAR, the PDF library
> from iText, the app server from Tomcat, write the servlet and I would
> NEVER ONCE have to deal with name clashes.
>
> Is this advantage so hard to appreciate?

I don't know in what form you get these libraries (e.g. source or 
binary), or what the integration process entails.  Certainly when we use 
pre-existing binary libraries (e.g. DLLs) naming isn't an issue.

> In the Forth world, management and developers would have to expend
> valuable effort managing this issue in addition to productive work like
> delivering functionality and eradicating bugs.

But, what Stephen and I, who have experience with a lot of applications 
of all sizes, are telling you that we *don't* expend "valuable effort" 
on this issue.

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Sat, 22 Oct 2011 15:19:43 +0000, Anton Ertl wrote:

> Arnold Doray <thinksquared@gmail.com> writes:
>>On Thu, 20 Oct 2011 13:19:46 +0000, Anton Ertl wrote:
>>
>>> Who knows?  In any case, it has not had an effect on the language, and
>>> that's despite the fact that name clashes are a much more severe
>>> problem in C than in Forth: If there is a name clash in C, the program
>>> won't build.  If there is a name clash in Forth, the program will
>>> usually compile and run as intended.
>>
>>At the risk of venturing into some serious speculation, I'd say it has
>>limited Forth's use in writing *larger* systems.
> 
> I doubt it.  The lack of a module system has not limited C's use in
> writing larger systems, like Netscape, the Linux kernel, Emacs, gcc.
> 

Good C uses far less names compared to Forth to accomplish the same task.

Also C has rudimentary methods of interface/implementation separation 
(header files), and exe's, dlls etc. What are the Forth equivalents? 

Also, what choice do these developers have in terms of language? The 
best, ubiquitous compilers are for C. What language you realistically 
choose to code an OS like Linux? I think they chose C *despite* its 
failings re. module support. They had little choice. 

Also, note that most of Emacs functionality is done in an embedded Lisp 
interpreter. 

>>I would argue that especially for Forth, successful "application"
>>programming requires solid module support to solve the name clash
>>problem.
>>
>>You can't compare "C" and Forth on this issue.
>>
>>"C" does not encourage factoring code to the extent that Forth does.
> 
> Ok.  So a program with 10,000 names is larger in idiomatic C than in
> idiomatic Forth.  Let's see:
> 
> [~/gforth/engine:76225] nm -A --defined-only dblsub.o io.o signals.o
> main-itc-noll.o |wc -l 136
> [~/gforth/engine:76226] wc -l dblsub.c io.c signals.c main.c
>     65 dblsub.c
>    764 io.c
>    462 signals.c
>   2442 main.c
>   3733 total
> 
> So we have about 27 lines per global symbol.
> 
> Now let's look at some Forth code:
> 
> [~/gforth:76257] gforth -e "hashpop ? cr bye" 2983
> [~/gforth:76258] gforth -i kernl64l.fi -e "words bye"|tr ' ' '\n'|tr -s
> '\n'|wc -l 1016
> 
> So, 2983-1016=1967 names outside the kernel.
> 
> [~/gforth:76260] wc -l `gforth -e ".included bye"` -e ".included bye"
>   ...
>   9090 total
> 
> So, about 4.6 lines per name.

OK, so all you've proven here is that C has more LOC per name compared to 
Forth. Not a novel insight surely? 

I'm saying that for the *same* task, Forth requires more names compared 
to C (although C may require many times more LOC). 

> 
> So if we are able to write 10M line systems in C (not sure how big the
> Linux kernel is at the moment), we should at least be able to write 1.7M
> line applications in Forth.

That's a big IF. Can you name a commonly known Forth app 10^6 LOC? (apart 
from Stephen's example)? For every such, I could name probably 50 in "C". 

> 
> And that's if name clashes would cause as big a problem in Forth as in
> C, which they don't.
> 

Right. See Stephen Pelc's experience with his 10^6 LOC app and 10^5 names 
and the variety of techniques MPE has to use to keep the hydra in check. 

> Of course, the idea of writing such large systems is considered horrible
> by a significant part of the Forth community.
> 

That's because a significant part of the Forth community are embedded 
developers, where large systems are anathema almost by definition. 

> 
> I think there are much larger impediments to the use of libraries and to
> writing large systems in Forth than name clash problems.  In particular,

Would you elaborate on these larger impediments? 

Cheers,
Arnold

On Oct 23, 10:57=A0am, BruceMcF <agil...@netscape.net> wrote:
> On Oct 23, 10:26=A0am, Krishna Myneni <krishna.myn...@ccreweb.org>
> wrote:
>
> > It's a tradeoff between the pain of using a specific modules system vs
> > resolving conflicts after the code has been written. The goal is to
> > permit all library modules to play together in whatever combination is
> > needed for an application. Consider that any time a new file is added
> > to the library, without a modules system, there's a potential for
> > breaking the user interface of any of the existing files. This is just
> > unwieldy to coordinate in a large distributed project. I would much
> > rather accept the minimal pain of using a modules system than
> > contending with the possibility of having to require authors to recode
> > their contributions every time new code is added.
>
> However, avoiding conflict with the public API of existing modules is
> a much less daunting task. A master list of the words forming the
> public API of each module, and a contributor can simply do a search
> for the words they propose to use to see if there are any conflicts.

Independent parallel development is one problem, as noted by Arnold.
However, what bothers me more, as someone who writes code, is the
potential restriction of not being able to use the most appropriate
and convenient names in my code. It is unnecessary to impose such a
restriction.

Krishna

On Mon, 24 Oct 2011 08:28:02 +0000 (UTC), Arnold Doray
<thinksquared@gmail.com> wrote:

>1) VFX Forth's own module system
>2) The app's internal module system
>3) Naming conventions
>4) Manual name clash resolution
>5) Wordlists
>6) Documentation/Culture ("discipline" ?)
>7) Active management oversight
>
>To a Forth newcomer who has had experience with other languages, this 
>seems pretty daunting. 

In reverse order:
a) All projects require discipline and project management,
   so 6) and 7) have little to do with name conflicts.
b) I did not mention wordlists, so remove 5).
c) 3) and 4) are the same.

This brings the list down to
1) VFX Forth's own module system
2) The app's internal module system
3) Naming conventions and nanual name clash resolution
4) Discipline and management

1) and 2) exist because the application's code base is far older than
the host Forth.

>Compare the situation in Java. There are a vast array of high quality 
>open source libraries available for it. As both an app developer and 
>library writer, I have never *ever* had to worry about name clashes. 

Java avoids this problem with name mangling and pushing complexity
into the linker. Mangled Java names of over 2000 characters have been
observed in the wild. See Uhlrich Drepper's "DSO how to" for the gory
details.

Forth name conflict is not really an issue, and simple module systems
fix the problem adequately. Programmers of large applications soon
learn not to use clever names - they know that words such as
DISPOSE have many meanings.

>Is this advantage so hard to appreciate? 

The problem it solves is way down my list of problems to be solved.

>But for app development, a solid module system is paramount for success. 

Despite the vast effort expended on c.l.f. recently, it should be
noted that some Forth systems, especially the commercial ones, have
had such systems for a very long time. A competent module system
with no frills takes about 20-30 lines of code.

Stephen


-- 
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

On Mon, 24 Oct 2011 09:48:35 +0000 (UTC), Arnold Doray
<thinksquared@gmail.com> wrote:

>Right. See Stephen Pelc's experience with his 10^6 LOC app and 10^5 names 
>and the variety of techniques MPE has to use to keep the hydra in check. 

There are no hydra.

Stephen


-- 
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

On Oct 24, 6:01=A0am, stephen...@mpeforth.com (Stephen Pelc) wrote:
> On Mon, 24 Oct 2011 08:28:02 +0000 (UTC), Arnold Doray

> >But for app development, a solid module system is paramount for success.
>
> Despite the vast effort expended on c.l.f. recently, it should be
> noted that some Forth systems, especially the commercial ones, have
> had such systems for a very long time. A competent module system
> with no frills takes about 20-30 lines of code.
>

And this really is the key point of the discussion. Module systems are
not hard to implement in Forth. Several in the Forth community have
disagreement on the design of such systems, not whether or not they
can be implemented. It's great if the problem has already been solved
for SwiftForth and for VFX . You are free to promote those over other
designs, since they can easily be implemented in Forth code to match
their described behavior. I (re)started the discussion in light of the
inadequate module system used in the FSL, and it evolved into a
discussion of a design that would be more widely applicable. The
SwiftForth and VFX module systems are such examples, but that doesn't
mean it's the end of the discussion.

Krishna



Krishna

stephenXXX@mpeforth.com (Stephen Pelc) writes:
>Java avoids this problem with name mangling and pushing complexity
>into the linker. Mangled Java names of over 2000 characters have been
>observed in the wild. See Uhlrich Drepper's "DSO how to" for the gory
>details.

You are probably thinking about C++.

Java has it's own object format (.class files) and its own linkage and
execution engine (the Java Virtual machine), so it does not need to
resort to name mangling, which is a mechanism that C++ uses to map
some of its special features onto (mostly) conventional object formats
and linkers.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

On Oct 24, 4:28=A0am, Arnold Doray <thinksqua...@gmail.com> wrote:
> On Sun, 23 Oct 2011 08:57:52 -0700, BruceMcF wrote:
>> However, avoiding conflict with the public API of existing modules is
>> a much less daunting task. A master list of the words forming the
>> public API of each module, and a contributor can simply do a search
>> for the words they propose to use to see if there are any conflicts.

> How would you coordinate public names for modules that are being
> developed independently?

> If you don't, then wouldn't the app developer have to bear the
> burden of resolving name conflicts of the libraries he uses
> (either through renaming, or changing the search order)?

Hypothetically, yes. However,

(1) if terse names assuming the problem context of the module are kept
to the PRIVATE section and the PUBLIC API words are more explicit,
then name conflicts will be relatively rare, if they crop up at all.
In particular, if using two libraries, only one of the two has to be
well behaved in that way to avoid name clashes.

(2) This is Forth. In 2011. Its not like there are going to be dozens
of competing source libraries out there.

Indeed, unlike dynamic link libraries, loading of selected modules out
of a source library is straightforward, which means that if a public
source library were to start to grow beyond the toy library stage, its
possible to cope with intrinsic implementation conflicts (option
support versus compiled code size, for instance) via including
alternative approaches rather than via forking.

(3) And further on (2), any public source library accepting
contributions that is being actively managed and promoted is likely to
be at FLAG, so with not much more trouble, so if there were three or
four libraries being actively managed and promoted, one could search
the master API list of all of them with not much more effort.

(4) And if its still an issue, one can always compile one of the
libraries that is the source of the conflict into its own vocabulary,
from the caller side, and the name conflicts go away. One vocabulary
per library and only if its _actually_ needed is much less trouble
than one PUBLIC API vocabulary per module where all of the
vocabularies may well be, in practice, an entirely redundant nuisance.

On Oct 24, 6:03=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:

> Independent parallel development is one problem, as noted by Arnold.
> However, what bothers me more, as someone who writes code, is the
> potential restriction of not being able to use the most appropriate
> and convenient names in my code. It is unnecessary to impose such a
> restriction.

For the Public API, the most appropriate name is one that is as clear
as practicable in as many application areas as practicable. That may
well not be the most convenient name inside the context of the module
lexicon ~ but even there, it will be the clearest.

That is, an author has to bear the intended audience in mind. The
intended audience for the Public API words are any possible user of
the module from the source library, and for that audience, a brief
name that has one connotation in the context of the module code but
other connotations in other contexts is first and foremost
inappropriate because it is so likely to be confusing.

Choosing appropriate names for public API words will, as a side
effect, tend to substantially reduce the prospect for name clashes.

For a module to be run in a variety of host lexicons, its a
convenience to the author to not have to think about that for each and
every word, so you only designate some words as part of the public
API, and are free to use personal shorthands and brief words that mean
one thing in this context and something else in some other context in
the Private section.

To go back to an example that has already been raised, if I have a
Maximum Entropy module that uses some other module (say, a simulated
annealing module) to arrive at the maximum entropy estimate of an
input-output table under a set of constraints, and there is a
tolerance for the underlying Entropy computation, and the tolerance
variable is part of the Public API, the primary reason I would not
call that "TOLERANCE" is that its *far less clear* than calling it
"Entropy-Tolerance". And given that there are other applications of
the Entropy statistic, its even clearer for someone who would be using
that MaxEnt module as a nonparametric estimator to call it "MaxEnt-
Tolerance".

On Oct 24, 7:05=A0am, stephen...@mpeforth.com (Stephen Pelc) wrote:
> On Mon, 24 Oct 2011 09:48:35 +0000 (UTC), Arnold Doray wrote:
>> Right. See Stephen Pelc's experience with his 10^6 LOC app and
>> 10^5 names and the variety of techniques MPE has to use to keep
>> the hydra in check.

> There are no hydra.

Pierson's Puppeteer? If the situation is one technique used to avoid
problems and fix any that occasionally arise, with two legacy systems
in place ~ why were the legacy systems in place in the first place?
Belts and suspenders? Training wheels?

On Mon, 24 Oct 2011 08:04:26 -0700, BruceMcF wrote:

>> How would you coordinate public names for modules that are being
>> developed independently?
> 
>> If you don't, then wouldn't the app developer have to bear the burden
>> of resolving name conflicts of the libraries he uses (either through
>> renaming, or changing the search order)?
> 
> Hypothetically, yes. However,
> 
> (1) if terse names assuming the problem context of the module are kept
> to the PRIVATE section and the PUBLIC API words are more explicit, then
> name conflicts will be relatively rare, if they crop up at all. In
> particular, if using two libraries, only one of the two has to be well
> behaved in that way to avoid name clashes.
>

Consider this: 

A and B are independently developed modules. 
Module C, independently developed, depends on modules A and B. 
Module C's developers notice name clashes in A and B and decide to rename 
those in A. 
Module D is yet another independent module, which also depends on A and 
B. 
D's developers decide to rename the functions in B to solve clashes. 

An app developer attempts to create an app using C and D, implicitly 
"requiring" A and B. 

What happens now? 

If your answer is that he manually resolves the conflicts himself, may I 
respectfully say that is the wrong answer. 

That is not all: Notice that if either A or B upgrade their 
implementation -- leaving the each API intact, then modules C, D and the 
app developer would *all* have to rename again. 

No app programmer in their right minds would choose a language -- in 2011 
-- that is that brittle. 

In Java, name clashes do not exist. Not "rarely happens". Does Not Exist. 

If your answer is that the scenario above is a gedakeneksperiment gone 
wild, then I would recommend you take a closer look at open source 
development, especially with Java, where most of it happens now.


> (2) This is Forth. In 2011. Its not like there are going to be dozens of
> competing source libraries out there.
> 
> Indeed, unlike dynamic link libraries, loading of selected modules out
> of a source library is straightforward, which means that if a public
> source library were to start to grow beyond the toy library stage, its
> possible to cope with intrinsic implementation conflicts (option support
> versus compiled code size, for instance) via including alternative
> approaches rather than via forking.


Actually, you may often want to implement the same API differently (I 
call it "interface", since API is usually only forward facing, while 
interfaces may be profitable between sub-modules) for a different reason 
than those you suggest above.

This is essentially the question of separating interface from 
implementation, which a module system should also address. 

For example, you might want to have a hash module, (the interface has 
"create" , "get" "put" , "remove" , "clear" , "size" and "dispose") but 
backed by either a red-black tree or a hash table or something else (eg, 
to efficiently serialize very large tables to disk, or to do it through 
RPC, or a hash table that loses entries if they are not read often 
enough). 

Same interface, different implementations. And not because there are 
"rival" implementations as you mean above. 

> 
> (3) And further on (2), any public source library accepting
> contributions that is being actively managed and promoted is likely to
> be at FLAG, so with not much more trouble, so if there were three or
> four libraries being actively managed and promoted, one could search the
> master API list of all of them with not much more effort.

No Bruce! That's a very narrow point of view. You should instead imagine 
a world in which Forth development is pervasive and build your module 
system from that perspective. 

> 
> (4) And if its still an issue, one can always compile one of the
> libraries that is the source of the conflict into its own vocabulary,
> from the caller side, and the name conflicts go away. One vocabulary per
> library and only if its _actually_ needed is much less trouble than one
> PUBLIC API vocabulary per module where all of the vocabularies may well
> be, in practice, an entirely redundant nuisance.

That sounds like a kludge. You probably felt a twinge of guilt just 
typing it yes?

Would it work well in the scenario I describe above? 

Cheers,
Arnold

On Mon, 24 Oct 2011 11:01:42 +0000, Stephen Pelc wrote:

> 
> This brings the list down to
> 1) VFX Forth's own module system
> 2) The app's internal module system
> 3) Naming conventions and nanual name clash resolution 4) Discipline and
> management
> 
> 1) and 2) exist because the application's code base is far older than
> the host Forth.
> 

Thank you for the clarification. 

I am beginning to see that the reason we see things disagree is probably 
due to a difference in worldviews. My basic concern is creating and using 
open source libraries and distributed application development. I want to 
see a thriving Forth open source community. 

This probably isn't high on your (or Elisabeth's) list of priorities, 
since your concern is primarily embedded development. Both worldviews are 
valid and have their place, and are complementary.

I've heard it said that Forth is not suited to application development. 
But I don't think that's true. For one, I do not get that impression 
reading articles when it was popular in the '80s and '90s. Also, I have 
used a number of languages, and I think Forth is very capable of this. 
But a solid module system, ie one that solves name clashes and interface/
implementation separation cleanly, is essential. 

It's true there aren't many Forth open source libraries now. But that 
need not remain the case. Even Ruby, which was created by one developer, 
has had open source libraries for everything from HTTPS sockets to big 
number support in just a few years. All done by volunteers. 

In other posts, I have already mentioned a possible new niche for Forth 
-- server side programs which are currently dominated by Java, C# and 
increasingly, Ruby. I have enough experience to say that I feel Forth can 
beat the *crap* out of these languages in this niche. It is just a much 
better language -- at least in this niche. I limit my statements because 
my experience is mainly in server programming.

But the large draw for Java and Ruby are their free, good, libraries. And 
the ease of creating and using libraries. 

This isn't just idealism. Especially for businesses like MPE and Forth 
Inc., there is much to be gained by fostering and contributing to open 
source development. I would highly recommend ESR's "Bazaar and Cathedral" 
book if you haven't already read it. 

> 
> Java avoids this problem with name mangling and pushing complexity into
> the linker. Mangled Java names of over 2000 characters have been
> observed in the wild. See Uhlrich Drepper's "DSO how to" for the gory
> details.
> 

You are probably thinking of C++.

Java solves the name clash issue using 2 ideas: All names are uniquely 
specified by a package structure. And any name must reside within a 
module (in Java, its a class). 

> fix the problem adequately. Programmers of large applications soon learn
> not to use clever names - they know that words such as DISPOSE have many
> meanings.
> 

That's my point. In your world, programmers must know where not to stand. 
In my world, programmers can stand anywhere they please. This may not 
seem to be a lot, but in my experience, it makes a huge difference in 
distributed programming. 

> 
> The problem it solves is way down my list of problems to be solved.
> 

Yes, I appreciate that. 

> 
> Despite the vast effort expended on c.l.f. recently, it should be noted
> that some Forth systems, especially the commercial ones, have had such
> systems for a very long time. A competent module system with no frills
> takes about 20-30 lines of code.
> 

The LOCs to do this is irrelevant. 

IMHO it is far more important to ensure that the proposed module system 
caters to the needs of the Forth community - both the embedded programmer 
(should she require one) and the app developer (who would most certainly 
want one). And it has to "feel" right. Anything less and it will not be 
widely adopted.

I think the discussion thus far has been diluted by bickering on 
implementation details. 

Perhaps the discussion should re-focus on just features/functionality?

Cheers,
Arnold

On Mon, 24 Oct 2011 18:59:08 +0000 (UTC), Arnold Doray
<thinksquared@gmail.com> wrote:

>I am beginning to see that the reason we see things disagree is probably 
>due to a difference in worldviews. My basic concern is creating and using 
>open source libraries and distributed application development. I want to 
>see a thriving Forth open source community. 
>
>This probably isn't high on your (or Elisabeth's) list of priorities, 
>since your concern is primarily embedded development. Both worldviews are 
>valid and have their place, and are complementary.

We've just been talking about 1 1M LOC commercial PC app!

I'll tell you what my primary concern is, not you me.

Open Source is just a different *business* model. The disciplines
required to generate the code are much the same. Whether the
programmers are all in the same building or scattered across 
several continents is an opportunity shared by both Open Source
and proprietary apps.

>In other posts, I have already mentioned a possible new niche for Forth 
>-- server side programs which are currently dominated by Java, C# and 
>increasingly, Ruby. I have enough experience to say that I feel Forth can 
>beat the *crap* out of these languages in this niche. It is just a much 
>better language -- at least in this niche. I limit my statements because 
>my experience is mainly in server programming.

Welcome to MPE's world - we have been doing server-side scripting in
Forth for over 10 years!

>But the large draw for Java and Ruby are their free, good, libraries. And 
>the ease of creating and using libraries. 

That's why MPE donates disk space to FLAG
  http://soton.mpeforth.com/flag/
and why I give time to it and other projects.

Stephen


-- 
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

On Oct 24, 10:34=A0am, BruceMcF <agil...@netscape.net> wrote:
> On Oct 24, 6:03=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
> > Independent parallel development is one problem, as noted by Arnold.
> > However, what bothers me more, as someone who writes code, is the
> > potential restriction of not being able to use the most appropriate
> > and convenient names in my code. It is unnecessary to impose such a
> > restriction.
>
> For the Public API, the most appropriate name is one that is as clear
> as practicable in as many application areas as practicable. That may
> well not be the most convenient name inside the context of the module
> lexicon ~ but even there, it will be the clearest.
>
> That is, an author has to bear the intended audience in mind. The
> intended audience for the Public API words are any possible user of
> the module from the source library, and for that audience, a brief
> name that has one connotation in the context of the module code but
> other connotations in other contexts is first and foremost
> inappropriate because it is so likely to be confusing.
>
> Choosing appropriate names for public API words will, as a side
> effect, tend to substantially reduce the prospect for name clashes.
>
> For a module to be run in a variety of host lexicons, its a
> convenience to the author to not have to think about that for each and
> every word, so you only designate some words as part of the public
> API, and are free to use personal shorthands and brief words that mean
> one thing in this context and something else in some other context in
> the Private section.
>
> To go back to an example that has already been raised, if I have a
> Maximum Entropy module that uses some other module (say, a simulated
> annealing module) to arrive at the maximum entropy estimate of an
> input-output table under a set of constraints, and there is a
> tolerance for the underlying Entropy computation, and the tolerance
> variable is part of the Public API, the primary reason I would not
> call that "TOLERANCE" is that its *far less clear* than calling it
> "Entropy-Tolerance". And given that there are other applications of
> the Entropy statistic, its even clearer for someone who would be using
> that MaxEnt module as a nonparametric estimator to call it "MaxEnt-
> Tolerance".


It's certainly true that an author given the freedom to assign names
as he/she pleases may, on first iteration, not necessarily choose the
most appropriate name for a public interface word. This is where the
library maintainer should suggest the author choose a more meaningful
name. But, such a suggestion should be for the sake of clarity to the
end users, not a strategy to avoid name clashes. Let's assume your
example in which the module's author names his public interface word
TOLERANCE. The end user decides that it makes more sense in his
application to name it "Entropy.Tolerance". The modules system permits
him to redefine these tolerances:

get-order
integrator  : integrator.tolerance tolerance ;
root-finder : root-finder.tolerance tolerance ;
MaxEnt      : entropy.tolerance tolerance ;
set-order

So the user can just as easily map the various tolerances to the names
which are meaningful in the application context. We are not imposing a
naming convention on the author, while still allowing a user to map
the module's public words to preferred names. Of course this can
always be done at INCLUDE time, but that can make for very ugly and
hard to follow code. Name remappings done in one section make it easy
to find such remappings.

Krishna

On 10/24/11 12:12 PM, Stephen Pelc wrote:
> On Mon, 24 Oct 2011 18:59:08 +0000 (UTC), Arnold Doray
> <thinksquared@gmail.com>  wrote:
>
>> I am beginning to see that the reason we see things disagree is probably
>> due to a difference in worldviews. My basic concern is creating and using
>> open source libraries and distributed application development. I want to
>> see a thriving Forth open source community.
>>
>> This probably isn't high on your (or Elisabeth's) list of priorities,
>> since your concern is primarily embedded development. Both worldviews are
>> valid and have their place, and are complementary.
>
> We've just been talking about 1 1M LOC commercial PC app!
>
> I'll tell you what my primary concern is, not you me.

Exactly.  I have no idea what Arnold's Forth experience level, but it 
does seem strange that so often here newbies are trying to explain why 
Forth "makes it hard" to do things that some of us have been doing with 
ease for many years. Forth itself is something of a different worldview. 
Many things that are issues in other languages, or for which elaborate 
mechanisms have proven necessary in other languages are simply not major 
issues in Forth.

> Open Source is just a different *business* model. The disciplines
> required to generate the code are much the same. Whether the
> programmers are all in the same building or scattered across
> several continents is an opportunity shared by both Open Source
> and proprietary apps.

Also true in my experience.

>> In other posts, I have already mentioned a possible new niche for Forth
>> -- server side programs which are currently dominated by Java, C# and
>> increasingly, Ruby. I have enough experience to say that I feel Forth can
>> beat the *crap* out of these languages in this niche. It is just a much
>> better language -- at least in this niche. I limit my statements because
>> my experience is mainly in server programming.
>
> Welcome to MPE's world - we have been doing server-side scripting in
> Forth for over 10 years!

So has FORTH, Inc. and several of our customers and colleagues.

>> But the large draw for Java and Ruby are their free, good, libraries. And
>> the ease of creating and using libraries.
>
> That's why MPE donates disk space to FLAG
>    http://soton.mpeforth.com/flag/
> and why I give time to it and other projects.

It's unfortunate that there are very few published libraries there. And 
I *don't* think it's because of people worried about name collisions! If 
people were eagerly posting libraries and users were complaining about 
name collisions, I'm sure we'd have heard about it!  In fact, I'm not at 
all sure whether Krishna got into this topic because of actual problems, 
or was just worrying about potential problems.

Introducing a note of reality here: if in 30 years piles of Forth 
libraries haven't been offered, and Java and Ruby and other languages 
dominate various their application domains, do you seriously think 
adding infrastructure that long-time professional Forth users don't want 
will suddenly change the world?

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Oct 24, 7:48=A0pm, "Elizabeth D. Rather" <erat...@forth.com> wrote:
....
> It's unfortunate that there are very few published libraries there. And
> I *don't* think it's because of people worried about name collisions! If
> people were eagerly posting libraries and users were complaining about
> name collisions, I'm sure we'd have heard about it! =A0In fact, I'm not a=
t
> all sure whether Krishna got into this topic because of actual problems,
> or was just worrying about potential problems.
>

The original problem was with the way in which the FSL implemented a
basic modules system. It simply reset the search order at the end of
each module which used the PUBLIC: / PRIVATE: mechanism. I had two
libraries on hand, a graphics library for X11 and the FSL. The Reset-
Search-Order caused the previously loaded X11 library to disappear
from the search order when FSL modules were loaded. While there is a
workaround for this, it showed that the module system in the FSL was
not well thought-out for use in larger applications. Replacing the
modular programming code in the FSL will solve the problem -- in fact,
I have already tested the module systems of Neal Bridges, and the one
I proposed here, and they both solve the problem in a satisfactory way
for the FSL. So, it's not just worrying about potential problems, but
trying to find the best solution to an existing problem which drives
this discussion.

At the very least we are now introduced to several modular programming
systems, including the ones employed by Forth Inc. and MPE. It's very
useful to know that large working applications have been realized with
such relatively simple facilities.


> Introducing a note of reality here: if in 30 years piles of Forth
> libraries haven't been offered, and Java and Ruby and other languages
> dominate various their application domains, do you seriously think
> adding infrastructure that long-time professional Forth users don't want
> will suddenly change the world?
>

Valid point. But, not giving proper weight to the infrastructure
issues which can help library development will kind of guarantee that
the status quo will not change.

The whole exercise of how to implement a modular programming system
points to a very useful feature missing in the current Forth standard.
That is, Forth-94 provides a very good set of words for handling the
search order and wordlist creation on which to base higher level
programming models such as modular code. However, we lack a portable
way to associate a name with a wordlist so that when the search order
is displayed, using ORDER , the name will appear in the list. I think
this needs to be addressed since it allows VOCABULARY to be defined in
a portable way using Forth source, as well as allowing more exotic
beasts such as vocabularies based on multiple wordlists. I will bring
this up again in a separate thread.

Regards,
Krishna

stephenXXX@mpeforth.com (Stephen Pelc) writes:
>Open Source is just a different *business* model. The disciplines
>required to generate the code are much the same.

Except where they are different.

For example, a program distributed in source and used by a lot of
people has to minimize the workload needed for installing it.  Among
other things, it has to work in a different directory from where it
was developed, without extra effort.  Now, proprietary code is usually
distributed to lots of people only in binary form, and only to few
people who develop and build it deal with the source, so someone used
to this distribution model thinks that asking the one using the source
code to change several lines in the source code to indicate the source
directory is ok.  But it isn't.  And when somebody comes up with a
proposal that solves the issue, they complain that it does not solve
the problem of finding the binary.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2011: http://www.euroforth.org/ef11/

On Mon, 24 Oct 2011 14:48:06 -1000, Elizabeth D. Rather wrote:

> Introducing a note of reality here: if in 30 years piles of Forth
> libraries haven't been offered, and Java and Ruby and other languages
> dominate various their application domains, do you seriously think
> adding infrastructure that long-time professional Forth users don't want
> will suddenly change the world?

That's an excellent observation. Have you asked yourself why is this so?  

Are you saying Forthers are less willing to contribute to open source 
compared to C/C++/Java/Ruby/Lisp/Haskell/OCaml programmers? Surely not.

Or is it a tacit admission that Forth just isn't well equipped to deal 
with these complexities? 

But I am optimistic. 

I believe addressing the infrastructure issues are a first step. Books 
that apply Forth in an *application* programming context would surely 
help to introduce Forth to a new generation of programmers. So would a 
"killer app" in Forth. Look at what Rails did for Ruby.

As for the validity of the problem, consider what people like Krishna who 
are actually contributing to open source libraries are saying. He's said 
that although he works alone, he wants the freedom to use the names he 
feels are suited to the problem, and not have to worry about name clash 
issues. I have brought up the same issue in a collaborative development 
environment. Other Forth veterans have also voiced the need to at least 
discuss modules too.

Of course, name clashes aren't everything! What is needed is a free 
module system that at a bare minimum definitively addresses name clash 
resolution and interface/implementation separation.

Kind Regards,
Arnold









 

On 25/10/2011 02:53, Krishna Myneni wrote:

[...]

> The whole exercise of how to implement a modular programming system
> points to a very useful feature missing in the current Forth standard.
> That is, Forth-94 provides a very good set of words for handling the
> search order and wordlist creation on which to base higher level
> programming models such as modular code. However, we lack a portable
> way to associate a name with a wordlist so that when the search order
> is displayed, using ORDER , the name will appear in the list.

Another barrier to portability that is related to use of modules is the 
lack of a portable way to specify directory paths. Suppose one module 
needs another, assumes it is in the same directory and includes it by:
    INCLUDE foo.fs
That is not guaranteed to work on all systems - it depends on the 
working directory and how a Forth system handles directory paths. There 
is not even a standard way to change a working directory.

This gets discussed periodically the last RfD I can find is 
https://groups.google.com/group/comp.lang.forth/browse_frm/thread/2e32aba5dd93fb3b/5d69974813ac673f?hl=en&lnk=gst&q=rfd+directory#5d69974813ac673f

but that doesn't address the use of libraries.

[...]

-- 
Gerry

Arnold Doray <thinksquared@gmail.com> wrote:
> On Mon, 24 Oct 2011 14:48:06 -1000, Elizabeth D. Rather wrote:
> 
>> Introducing a note of reality here: if in 30 years piles of Forth
>> libraries haven't been offered, and Java and Ruby and other languages
>> dominate various their application domains, do you seriously think
>> adding infrastructure that long-time professional Forth users don't want
>> will suddenly change the world?
> 
> That's an excellent observation. Have you asked yourself why is this so?  
> 
> Are you saying Forthers are less willing to contribute to open
> source compared to C/C++/Java/Ruby/Lisp/Haskell/OCaml programmers?
> Surely not.

Probably.  A lot of Forth's strengths are in embedded applications,
where open source has never been a strong force.  There isn't a huge
number of Forth programmers working in public.

> Or is it a tacit admission that Forth just isn't well equipped to
> deal with these complexities?

No.

> Of course, name clashes aren't everything! What is needed is a free
> module system that at a bare minimum definitively addresses name
> clash resolution and interface/implementation separation.

The really strange thing here is that even testimony from the leading
Forth practitioners (and yes, they certainly are) makes no difference
to your opinion about what is needed, despite the fact that you
describe yourself as a "Forth newcomer".

Andrew.

On Oct 24, 2:59=A0pm, Arnold Doray <thinksqua...@gmail.com> wrote:
> On Mon, 24 Oct 2011 08:04:26 -0700, BruceMcF wrote:
> >> How would you coordinate public names for modules that are being
> >> developed independently?
>
> >> If you don't, then wouldn't the app developer have to bear the burden
> >> of resolving name conflicts of the libraries he uses (either through
> >> renaming, or changing the search order)?
>
> > Hypothetically, yes. However,

> > (1) if terse names assuming the problem context of the module are kept
> > to the PRIVATE section and the PUBLIC API words are more explicit, then
> > name conflicts will be relatively rare, if they crop up at all. In
> > particular, if using two libraries, only one of the two has to be well
> > behaved in that way to avoid name clashes.

> Consider this:

> A and B are independently developed modules.
> Module C, independently developed, depends on modules A and B.
> Module C's developers notice name clashes in A and B and decide to rename
> those in A.

> Module D is yet another independent module, which also depends on A and
> B.
> D's developers decide to rename the functions in B to solve clashes.

> An app developer attempts to create an app using C and D, implicitly
> "requiring" A and B.

> What happens now?

*Now*, when he loads C into one vocabulary and D into another
vocabulary, A and B both get loaded twice, together with different
renaming of the clashing API words in each vocabulary. If there is no
clash between the C API and the D API, the first can be loaded into a
side vocabulary, with bridge definitions in the same name added to the
main vocabulary, then second loaded into the main vocabulary.

> That is not all: Notice that if either A or B upgrade their
> implementation -- leaving the each API intact, then modules C, D and the
> app developer would *all* have to rename again.

If the API is intact, there is no need to redefine the bridge
definition code in either module C or module D, it just needs to be
run with the new implementation.

> No app programmer in their right minds would choose a language -- in 2011
> -- that is that brittle.

But you've described both module C and module D doing things blindly,
and the brittleness is that modules A and B get loaded twice.

> In Java, name clashes do not exist. Not "rarely happens". Does Not Exist.

> If your answer is that the scenario above is a gedakeneksperiment gone
> wild, then I would recommend you take a closer look at open source
> development, especially with Java, where most of it happens now.

My answer is you assumed that there is a functional problem using both
C and D side by side, when the problem is rather efficiency ~ the
dictionary bloat from loading the same modules twice.

> > (2) This is Forth. In 2011. Its not like there are going to be dozens o=
f
> > competing source libraries out there.

> > Indeed, unlike dynamic link libraries, loading of selected modules out
> > of a source library is straightforward, which means that if a public
> > source library were to start to grow beyond the toy library stage, its
> > possible to cope with intrinsic implementation conflicts (option suppor=
t
> > versus compiled code size, for instance) via including alternative
> > approaches rather than via forking.

> Actually, you may often want to implement the same API differently (I
> call it "interface", since API is usually only forward facing, while
> interfaces may be profitable between sub-modules) for a different reason
> than those you suggest above.

> This is essentially the question of separating interface from
> implementation, which a module system should also address.

> For example, you might want to have a hash module, (the interface has
> "create" , "get" "put" , "remove" , "clear" , "size" and "dispose") but
> backed by either a red-black tree or a hash table or something else (eg,
> to efficiently serialize very large tables to disk, or to do it through
> RPC, or a hash table that loses entries if they are not read often
> enough).

> Same interface, different implementations. And not because there are
> "rival" implementations as you mean above.

What is the substantial point here? There are a variety of existing
mechanisms for seperating interface from implementation in Forth. You
are arging that the module system should supplant those? Should
cooperate with those? Should be compatible with those?

> > (3) And further on (2), any public source library accepting
> > contributions that is being actively managed and promoted is likely to
> > be at FLAG, so with not much more trouble, so if there were three or
> > four libraries being actively managed and promoted, one could search th=
e
> > master API list of all of them with not much more effort.

> No Bruce! That's a very narrow point of view. You should instead imagine
> a world in which Forth development is pervasive and build your module
> system from that perspective.

A mighty oak may have grown from a tiny acorn, but it had to survive
as a sapling to get there. The more hypothetical problems not
presently experienced you solve, the further in advance (1) the more
you burden current contributors with jumping through the hypothetical
problem solving hoops without the benefit of participating without an
already substantial collection of libraries of modules to justify
jumping through those hoops and (2) the more likely you are going to
channel development according to something in your vision that is
missing from the reality if it occurs, and away from something missing
from your vision that is present in the reality if it occurs.

> > (4) And if its still an issue, one can always compile one of the
> > libraries that is the source of the conflict into its own vocabulary,
> > from the caller side, and the name conflicts go away. One vocabulary pe=
r
> > library and only if its _actually_ needed is much less trouble than one
> > PUBLIC API vocabulary per module where all of the vocabularies may well
> > be, in practice, an entirely redundant nuisance.

> That sounds like a kludge. You probably felt a twinge of guilt just
> typing it yes?

No, that is what vocabularies are for.

> Would it work well in the scenario I describe above?

It certainly works, without name clashes, though it can be downgraded
for dictionary bloat. The refined solutions is for someone to
contribute a joint A&B loadscript that creates an integrated joint
API, and for C and D to adopt the joint A&B loadscript.

On Oct 24, 7:34=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
> On Oct 24, 10:34=A0am, BruceMcF <agil...@netscape.net> wrote:
>
>
>
>
>
> > On Oct 24, 6:03=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wrote=
:
>
> > > Independent parallel development is one problem, as noted by Arnold.
> > > However, what bothers me more, as someone who writes code, is the
> > > potential restriction of not being able to use the most appropriate
> > > and convenient names in my code. It is unnecessary to impose such a
> > > restriction.
>
> > For the Public API, the most appropriate name is one that is as clear
> > as practicable in as many application areas as practicable. That may
> > well not be the most convenient name inside the context of the module
> > lexicon ~ but even there, it will be the clearest.
>
> > That is, an author has to bear the intended audience in mind. The
> > intended audience for the Public API words are any possible user of
> > the module from the source library, and for that audience, a brief
> > name that has one connotation in the context of the module code but
> > other connotations in other contexts is first and foremost
> > inappropriate because it is so likely to be confusing.
>
> > Choosing appropriate names for public API words will, as a side
> > effect, tend to substantially reduce the prospect for name clashes.
>
> > For a module to be run in a variety of host lexicons, its a
> > convenience to the author to not have to think about that for each and
> > every word, so you only designate some words as part of the public
> > API, and are free to use personal shorthands and brief words that mean
> > one thing in this context and something else in some other context in
> > the Private section.
>
> > To go back to an example that has already been raised, if I have a
> > Maximum Entropy module that uses some other module (say, a simulated
> > annealing module) to arrive at the maximum entropy estimate of an
> > input-output table under a set of constraints, and there is a
> > tolerance for the underlying Entropy computation, and the tolerance
> > variable is part of the Public API, the primary reason I would not
> > call that "TOLERANCE" is that its *far less clear* than calling it
> > "Entropy-Tolerance". And given that there are other applications of
> > the Entropy statistic, its even clearer for someone who would be using
> > that MaxEnt module as a nonparametric estimator to call it "MaxEnt-
> > Tolerance".
>
> It's certainly true that an author given the freedom to assign names
> as he/she pleases may, on first iteration, not necessarily choose the
> most appropriate name for a public interface word. This is where the
> library maintainer should suggest the author choose a more meaningful
> name. But, such a suggestion should be for the sake of clarity to the
> end users, not a strategy to avoid name clashes.

Yes. TOLERANCE used as a specific variable for a specific procedure is
quite obviously obscure. If the module is going to use TOLERANCE and
DELTA and EPSILON as public API names, those *obviously* should go
into a named vocabulary.

But why does that mean *all* the public API should go into a named
vocabulary? It doesn't follow.

Module-Name: IO-MaxEnt-Estimate
....
Begin-Module
Export-To: IO-Maxent-Settings

EXPORT

FVARIABLE TOLERANCE
FVARIABLE DELTA
FVARIABLE EPSILON

PRIVATE
  ...
PUBLIC
  ...
End-Module

Now you've got your TOLERANCE hidden away in a vocabulary because *it*
is guilty of being a deliberately obscure name if taken out of
context, when (presumably) most of your API will not be named in such
a deliberately obscure manner.

> Let's assume your
> example in which the module's author names his public interface word
> TOLERANCE.

> The end user decides that it makes more sense in his
> application to name it "Entropy.Tolerance". The modules system permits
> him to redefine these tolerances:

> get-order
> integrator =A0: integrator.tolerance tolerance ;
> root-finder : root-finder.tolerance tolerance ;
> MaxEnt =A0 =A0 =A0: entropy.tolerance tolerance ;
> set-order

But you're arguing in circles if you are arguing that the Public API
words must all be a vocabulary named after the module, on the basis of
name choices for the Public API that can only be justified because the
Public API words are all in a seperate vocabulary.

> So the user can just as easily map the various tolerances to the names
> which are meaningful in the application context. We are not imposing a
> naming convention on the author, while still allowing a user to map
> the module's public words to preferred names.

But you are taking away the existing control over where the Public API
words go.

> Of course this can
> always be done at INCLUDE time, but that can make for very ugly and
> hard to follow code. Name remappings done in one section make it easy
> to find such remappings.

At the cost of forcing everyone who wants to group the API from
multiple modules into the same lexicon of doing renamings that would
normally be unecessary.

In article <j84chr$u8j$1@dont-email.me>,
Arnold Doray  <thinksquared@gmail.com> wrote:
>On Mon, 24 Oct 2011 08:04:26 -0700, BruceMcF wrote:
>
>>> How would you coordinate public names for modules that are being
>>> developed independently?
>>
>>> If you don't, then wouldn't the app developer have to bear the burden
>>> of resolving name conflicts of the libraries he uses (either through
>>> renaming, or changing the search order)?
>>
>> Hypothetically, yes. However,
>>
>> (1) if terse names assuming the problem context of the module are kept
>> to the PRIVATE section and the PUBLIC API words are more explicit, then
>> name conflicts will be relatively rare, if they crop up at all. In
>> particular, if using two libraries, only one of the two has to be well
>> behaved in that way to avoid name clashes.
>>
>
>Consider this:
>
>A and B are independently developed modules.
>Module C, independently developed, depends on modules A and B.
>Module C's developers notice name clashes in A and B and decide to rename
>those in A.
>Module D is yet another independent module, which also depends on A and
>B.
>D's developers decide to rename the functions in B to solve clashes.
>
>An app developer attempts to create an app using C and D, implicitly
>"requiring" A and B.
>
>What happens now?
>
>If your answer is that he manually resolves the conflicts himself, may I
>respectfully say that is the wrong answer.
>
>That is not all: Notice that if either A or B upgrade their
>implementation -- leaving the each API intact, then modules C, D and the
>app developer would *all* have to rename again.
>
>No app programmer in their right minds would choose a language -- in 2011
>-- that is that brittle.
>
>In Java, name clashes do not exist. Not "rarely happens". Does Not Exist.

That sounds like important. Currently I'm working on a large
Java application in a team. I imagine an alternative in Forth.
In our Java environment I see hashed name tables that would be
so much easier in Forth. I could present this aspect in a
way that Java looks very bad and Forth very good.

>
>If your answer is that the scenario above is a gedakeneksperiment gone
>wild, then I would recommend you take a closer look at open source
>development, especially with Java, where most of it happens now.
>
>
>> (2) This is Forth. In 2011. Its not like there are going to be dozens of
>> competing source libraries out there.
>>
>> Indeed, unlike dynamic link libraries, loading of selected modules out
>> of a source library is straightforward, which means that if a public
>> source library were to start to grow beyond the toy library stage, its
>> possible to cope with intrinsic implementation conflicts (option support
>> versus compiled code size, for instance) via including alternative
>> approaches rather than via forking.
>
>
>Actually, you may often want to implement the same API differently (I
>call it "interface", since API is usually only forward facing, while
>interfaces may be profitable between sub-modules) for a different reason
>than those you suggest above.
>
>This is essentially the question of separating interface from
>implementation, which a module system should also address.

A lot of academic theory. I can't say my 30+ years of experience
corroborates this much.

>
>For example, you might want to have a hash module, (the interface has
>"create" , "get" "put" , "remove" , "clear" , "size" and "dispose") but
>backed by either a red-black tree or a hash table or something else (eg,
>to efficiently serialize very large tables to disk, or to do it through
>RPC, or a hash table that loses entries if they are not read often
>enough).

A text book example. A Dutch Forther (Albert Nijhof) has a perfect
solution for this. Guess what? It is not used often. Guess why?
It is not needed often.

>
>Same interface, different implementations. And not because there are
>"rival" implementations as you mean above.
>
>>
>> (3) And further on (2), any public source library accepting
>> contributions that is being actively managed and promoted is likely to
>> be at FLAG, so with not much more trouble, so if there were three or
>> four libraries being actively managed and promoted, one could search the
>> master API list of all of them with not much more effort.
>
>No Bruce! That's a very narrow point of view. You should instead imagine
>a world in which Forth development is pervasive and build your module
>system from that perspective.
>
>>
>> (4) And if its still an issue, one can always compile one of the
>> libraries that is the source of the conflict into its own vocabulary,
>> from the caller side, and the name conflicts go away. One vocabulary per
>> library and only if its _actually_ needed is much less trouble than one
>> PUBLIC API vocabulary per module where all of the vocabularies may well
>> be, in practice, an entirely redundant nuisance.
>
>That sounds like a kludge. You probably felt a twinge of guilt just
>typing it yes?
>
>Would it work well in the scenario I describe above?

I can answer that but I will let Bruce talk for himself.

>
>Cheers,
>Arnold
>

Groetjes Albert


--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

On Oct 24, 2:59=A0pm, Arnold Doray <thinksqua...@gmail.com> wrote:
> A and B are independently developed modules.
> Module C, independently developed, depends on modules A and B.
> Module C's developers notice name clashes in A and B and decide to rename=
 those in A.

I want to be clear here ~ when you say "decide to rename those in A",
you mean to define a bridge module with new definitions that are
defined into term of the "clashed" module?

That is, they have a loadscript:

-------------------
\ afoo-renamed.fs

INCLUDE afoo.fs

Module-Name: afoo-renamed
Import-Public: afoo
PUBLIC
: afoo-bar1 bar1 ;
: afoo-bar2 bar2 ;

[has?] Public-Wordlists? [IF]
   : .afoo .afoo ;
   : }}afoo }}afoo ;
   : }afoo }afoo ;
[THEN]
End-Module
------------------

And module C is set up as:

-------------------
REQUIRE afoo-renamed.fs
REQUIRE bfoo.fs

Name-Module: cfoo
Import-Public: afoo-renamed
Import-Public: bfoo
Begin-Module
 ...
End-Module

You surely don't mean that Module C Developers *edit* Module A to have
the names they prefer? If they do *that*, and the app developer wants
to use it anyway ... well, if they are using a shonky module, its
because its got some sweet code in it somewhere, but I think the way
to go would be to copy the sweet code and write a a non-shonky version
of Module C around it, and then you have a non-shonky version of
Module C to use from then on.

On 10/25/11 11:14 AM, Arnold Doray wrote:
> On Mon, 24 Oct 2011 14:48:06 -1000, Elizabeth D. Rather wrote:
>
>> Introducing a note of reality here: if in 30 years piles of Forth
>> libraries haven't been offered, and Java and Ruby and other languages
>> dominate various their application domains, do you seriously think
>> adding infrastructure that long-time professional Forth users don't want
>> will suddenly change the world?
>
> That's an excellent observation. Have you asked yourself why is this so?

Often.  And also discussed it with Forthers and lots of folks in the 
business.

> Are you saying Forthers are less willing to contribute to open source
> compared to C/C++/Java/Ruby/Lisp/Haskell/OCaml programmers? Surely not.

In the first place, there are fewer of them.  Forth hasn't been picked 
up and promoted in the universities.  But, also, they tend to focus on 
specific solutions to specific problems (i.e., the project that's paying 
the bills right now), and oftener than not, that project is proprietary 
and the company paying the bills isn't keen for the code to be shared.

> Or is it a tacit admission that Forth just isn't well equipped to deal
> with these complexities?

No.

> But I am optimistic.
>
> I believe addressing the infrastructure issues are a first step. Books
> that apply Forth in an *application* programming context would surely
> help to introduce Forth to a new generation of programmers. So would a
> "killer app" in Forth. Look at what Rails did for Ruby.

Ruby on Rails was extracted by David Heinemeier Hansson from his work on 
Basecamp, a project management tool by 37signals (now a web application 
company). From the look of his web site, he's doing well with it.  But 
it was developed and extensively promoted by someone willing and able to 
pay the bills, and both are expensive propositions.  It's mostly the 
promotion budget that's been missing in Forth, not module infrastructure.

> As for the validity of the problem, consider what people like Krishna who
> are actually contributing to open source libraries are saying. He's said
> that although he works alone, he wants the freedom to use the names he
> feels are suited to the problem, and not have to worry about name clash
> issues. I have brought up the same issue in a collaborative development
> environment. Other Forth veterans have also voiced the need to at least
> discuss modules too.

Sure. And Forth, Inc. and MPE and others have developed module solutions 
when needed.  But that has microscopically little to do with the lack of 
Forth libraries.

> Of course, name clashes aren't everything! What is needed is a free
> module system that at a bare minimum definitively addresses name clash
> resolution and interface/implementation separation.

No, what's needed are people who have appropriate economic support with 
which to develop significant bodies of code that can be released to the 
public as open source, plus a promotion budget that lets the world know 
that they exist.

This is not a technical problem.

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On 10/25/11 6:22 AM, Anton Ertl wrote:
> stephenXXX@mpeforth.com (Stephen Pelc) writes:
>> Open Source is just a different *business* model. The disciplines
>> required to generate the code are much the same.
>
> Except where they are different.
>
> For example, a program distributed in source and used by a lot of
> people has to minimize the workload needed for installing it.  Among
> other things, it has to work in a different directory from where it
> was developed, without extra effort.  Now, proprietary code is usually
> distributed to lots of people only in binary form, and only to few
> people who develop and build it deal with the source, so someone used
> to this distribution model thinks that asking the one using the source
> code to change several lines in the source code to indicate the source
> directory is ok.  But it isn't.  And when somebody comes up with a
> proposal that solves the issue, they complain that it does not solve
> the problem of finding the binary.

If you're talking about FORTH, Inc. and MPE, their proprietary code is 
released in source to everyone who pays the license fee. If you're 
talking about the proprietary code that companies pay FORTH, Inc., MPE, 
and large numbers of other Forth programmers to develop, those companies 
have every right to look at what they paid to have this code developed 
and consider what business model would make it productive to release and 
promote it.

Cheers,
Elizabeth

-- 
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Oct 25, 12:44=A0pm, BruceMcF <agil...@netscape.net> wrote:
> On Oct 24, 7:34=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:
>
>
>
>
>
>
>
>
>
> > On Oct 24, 10:34=A0am, BruceMcF <agil...@netscape.net> wrote:
>
> > > On Oct 24, 6:03=A0am, Krishna Myneni <krishna.myn...@ccreweb.org> wro=
te:
>
> > > > Independent parallel development is one problem, as noted by Arnold=
..
> > > > However, what bothers me more, as someone who writes code, is the
> > > > potential restriction of not being able to use the most appropriate
> > > > and convenient names in my code. It is unnecessary to impose such a
> > > > restriction.
>
> > > For the Public API, the most appropriate name is one that is as clear
> > > as practicable in as many application areas as practicable. That may
> > > well not be the most convenient name inside the context of the module
> > > lexicon ~ but even there, it will be the clearest.
>
> > > That is, an author has to bear the intended audience in mind. The
> > > intended audience for the Public API words are any possible user of
> > > the module from the source library, and for that audience, a brief
> > > name that has one connotation in the context of the module code but
> > > other connotations in other contexts is first and foremost
> > > inappropriate because it is so likely to be confusing.
>
> > > Choosing appropriate names for public API words will, as a side
> > > effect, tend to substantially reduce the prospect for name clashes.
>
> > > For a module to be run in a variety of host lexicons, its a
> > > convenience to the author to not have to think about that for each an=
d
> > > every word, so you only designate some words as part of the public
> > > API, and are free to use personal shorthands and brief words that mea=
n
> > > one thing in this context and something else in some other context in
> > > the Private section.
>
> > > To go back to an example that has already been raised, if I have a
> > > Maximum Entropy module that uses some other module (say, a simulated
> > > annealing module) to arrive at the maximum entropy estimate of an
> > > input-output table under a set of constraints, and there is a
> > > tolerance for the underlying Entropy computation, and the tolerance
> > > variable is part of the Public API, the primary reason I would not
> > > call that "TOLERANCE" is that its *far less clear* than calling it
> > > "Entropy-Tolerance". And given that there are other applications of
> > > the Entropy statistic, its even clearer for someone who would be usin=
g
> > > that MaxEnt module as a nonparametric estimator to call it "MaxEnt-
> > > Tolerance".
>
> > It's certainly true that an author given the freedom to assign names
> > as he/she pleases may, on first iteration, not necessarily choose the
> > most appropriate name for a public interface word. This is where the
> > library maintainer should suggest the author choose a more meaningful
> > name. But, such a suggestion should be for the sake of clarity to the
> > end users, not a strategy to avoid name clashes.
>
> Yes. TOLERANCE used as a specific variable for a specific procedure is
> quite obviously obscure. ...

Obscure to whom? In the context of a particular module, it should be
perfectly clear as to what it represents. Otherwise, a better name
should be chosen.

> If the module is going to use TOLERANCE and
> DELTA and EPSILON as public API names, those *obviously* should go
> into a named vocabulary.
>

Yes, these are perfectly common names with common meanings in
numerical computing. Within a specific problem area, such as finding
roots or eigenvalues, it's superfluous to assert that an author should
impose prefixes to these names, for the sake of avoiding name clashes.

> But why does that mean *all* the public API should go into a named
> vocabulary? It doesn't follow.
>

I'm not against allowing the public API words to go into the existing
compilation wordlist, such as the Forth wordlist. This is why
anonymous modules still exist in my current implementation. It seems
like it would be unwieldy though to have the public interface words
distributed among more than one wordlist. In my current scheme,
derived from DNWs root-module.fs, invoking the module name brings
*all* of the public interface words into the search order.

> Module-Name: IO-MaxEnt-Estimate
> ...
> Begin-Module
> Export-To: IO-Maxent-Settings
>
> EXPORT
>
> FVARIABLE TOLERANCE
> FVARIABLE DELTA
> FVARIABLE EPSILON
>
> PRIVATE
> =A0 ...
> PUBLIC
> =A0 ...
> End-Module
>
> Now you've got your TOLERANCE hidden away in a vocabulary because *it*
> is guilty of being a deliberately obscure name if taken out of
> context, when (presumably) most of your API will not be named in such
> a deliberately obscure manner.
>

I don't follow your assertion that the name TOLERANCE is deliberately
obscure. This is your example, with regards to entropy, and I don't
understand what TOLERANCE is supposed to mean in that context. But
presumably, you as the author chose that name to be meaningful to
yourself and to your intended users, within the context of your
module. If not, choose a better name. Also, I don't understand what
are the intended functions of EXPORT-TO: and EXPORT .


> > Let's assume your
> > example in which the module's author names his public interface word
> > TOLERANCE.
> > The end user decides that it makes more sense in his
> > application to name it "Entropy.Tolerance". The modules system permits
> > him to redefine these tolerances:
> > get-order
> > integrator =A0: integrator.tolerance tolerance ;
> > root-finder : root-finder.tolerance tolerance ;
> > MaxEnt =A0 =A0 =A0: entropy.tolerance tolerance ;
> > set-order
>
> But you're arguing in circles if you are arguing that the Public API
> words must all be a vocabulary named after the module, on the basis of
> name choices for the Public API that can only be justified because the
> Public API words are all in a seperate vocabulary.
>

Not at all. The argument I'm making is that the author of a module is
free to choose meaningful names for his/her API *without regard* to
name clashes. That's the essential reason for using a separate
wordlist for the API. If the application programmer decides those
names are not the best choice for whatever reason, separate names
*can* be assigned to them. For example, if several public API words
from different modules have the same name, the user may not want to
fiddle with the search order to access a particular module's word
within the middle of a word definition. Then, the user can create
names which are meaningful to him, within a single block of code. The
practice of using *preassigned* name prefixes will certainly
accomplish the same thing, but that's a one size fits all approach
that leads to hard to read code. It's a practice that's been rejected
in modern programming languages, presumably not because it's just a
fashionable thing to do but because it leads to real benefits in code
comprehensibility and maintenance.


> > So the user can just as easily map the various tolerances to the names
> > which are meaningful in the application context. We are not imposing a
> > naming convention on the author, while still allowing a user to map
> > the module's public words to preferred names.
>
> But you are taking away the existing control over where the Public API
> words go.
>

I really don't understand this statement. Yes, the public API goes
into a different wordlist. So what? It's easily placed into the search
order by invoking the module's name. What difference does it make in
which wordlist a word appears if the word is in the search order?
That's really all that the user cares about when using the API words,
isn't it?

> > Of course this can
> > always be done at INCLUDE time, but that can make for very ugly and
> > hard to follow code. Name remappings done in one section make it easy
> > to find such remappings.
>
> At the cost of forcing everyone who wants to group the API from
> multiple modules into the same lexicon of doing renamings that would
> normally be unecessary.

No one is being forced to do anything. The provision exists to place
the API words from multiple modules into the same lexicon -- one can
use anonymous modules and avoid name clashes with the "disciplined",
brute-force approach then. I do agree that even for anonymous modules,
the private words should go into an accessible wordlist, as opposed to
disappearing from user view for good.

Krishna

On Oct 25, 9:40=A0pm, Krishna Myneni <krishna.myn...@ccreweb.org> wrote:

> Obscure to whom? In the context of a particular module, it should be
> perfectly clear as to what it represents. Otherwise, a better name
> should be chosen.

Obscure to users who minds are not *in* the context of the module, but
rather whose minds are in the context of *their* problem.


> > If the module is going to use TOLERANCE and
> > DELTA and EPSILON as public API names, those *obviously* should go
> > into a named vocabulary.

> Yes, these are perfectly common names with common meanings in
> numerical computing. Within a specific problem area, such as finding
> roots or eigenvalues, it's superfluous to assert that an author should
> impose prefixes to these names, for the sake of avoiding name clashes.

But they are first and foremost obscure as words in general.
"tolerance of what?", "delta of what?" *Setting aside the issue of
name clashes*, if they are part of the public API, they are not as
self-explanatory as they could easily be made to be.

> > But why does that mean *all* the public API should go into a named
> > vocabulary? It doesn't follow.
>
> I'm not against allowing the public API words to go into the existing
> compilation wordlist, such as the Forth wordlist. This is why
> anonymous modules still exist in my current implementation.

Where I'm against that implementation is that whether there should be
a module name and whether the PUBLIC words should go into distinct
vocabularies are two distinct issues, and it unnecessarily constrains
the functionality of the traditional PUBLIC behavior to conflate it
with anonymous modules.

> It seems
> like it would be unwieldy though to have the public interface words
> distributed among more than one wordlist.

The best way to manage the named public vocabularies system is to
compile *them* all into a single wordlist, whether FORTH or LIBRARY or
whatever. So working with them is *always* a matter of being in the
base vocabulary and then opening up side vocabularies.

The approach of *only* putting words into a side vocabulary in the
event that the selective visibility is called for has *the same* base
wordlist, and *far fewer* side vocabularies. So, (1) when the side
vocabularies do not need to be opened up, it is less unwieldy, (2) if
the side vocabularies need to be opened up, it is no more unwieldy.


> In my current scheme,
> derived from DNWs root-module.fs, invoking the module name brings
> *all* of the public interface words into the search order.

Yes, and DNW's root-module.fs is an experimental module system, while
the tried and tested module systems do not do it that way.

> > Module-Name: IO-MaxEnt-Estimate
> > ...
> > Begin-Module
> > Export-To: IO-Maxent-Settings
>
> > EXPORT
>
> > FVARIABLE TOLERANCE
> > FVARIABLE DELTA
> > FVARIABLE EPSILON
>
> > PRIVATE
> > =A0 ...
> > PUBLIC
> > =A0 ...
&g