Hello,

I've written an IOLib tutorial which describes how to write network
clients and servers with IPV4 TCP using blocking I/O with the stream
interface and nonblocking I/O with the event multiplexer also including
send-to and receive-from.

The tutorial and associated example program sources for which it
is written are in the examples/ directory of the IOLib source codes.
The tutorial and source code in the official source release of IOLib will
always be the authoritative version. It is recommended one inspects and
tries out the example source code along with the tutorial.

The latest version of IOLib is always under "Live Sources" on this web
page: 

http://common-lisp.net/project/iolib/download.shtml

The repository mentioned there is:                  

git clone git://gitorious.org/iolib/iolib.git

If you use clbuild to manage your lisp environment then (while making
sure your clbuild is up to date):

../clbuild install iolib

will get you the latest revision of the example programs and tutorial.

I welcome additional example codes and patches to the tutorial and
will help test the code and document it into the tutorial. Any bugs
or changes to IOLib itself should be directed to Stelian Ionescu--the
author of IOLib.

Here is a link to a very recent edition of the tutorial:

http://pages.cs.wisc.edu/~psilord/blog/data/iolib-tutorial/tutorial.html

Thank you.

Peter Keller

On 2010-05-29 06:30 +0100, Peter Keller wrote:
> Here is a link to a very recent edition of the tutorial:
>
> http://pages.cs.wisc.edu/~psilord/blog/data/iolib-tutorial/tutorial.html
>
> Thank you.

Thanks.

Leo

Peter Keller wrote:
> Hello,
> 
> I've written an IOLib tutorial which describes how to write network
> clients and servers with IPV4 TCP using blocking I/O with the stream
> interface and nonblocking I/O with the event multiplexer also including
> send-to and receive-from.
> 
> The tutorial and associated example program sources for which it
> is written are in the examples/ directory of the IOLib source codes.
> The tutorial and source code in the official source release of IOLib will
> always be the authoritative version. It is recommended one inspects and
> tries out the example source code along with the tutorial.
> 
> The latest version of IOLib is always under "Live Sources" on this web
> page: 
> 
> http://common-lisp.net/project/iolib/download.shtml
> 
> The repository mentioned there is:                  
> 
> git clone git://gitorious.org/iolib/iolib.git
> 
> If you use clbuild to manage your lisp environment then (while making
> sure your clbuild is up to date):
> 
> ./clbuild install iolib
> 
> will get you the latest revision of the example programs and tutorial.
> 
> I welcome additional example codes and patches to the tutorial and
> will help test the code and document it into the tutorial. Any bugs
> or changes to IOLib itself should be directed to Stelian Ionescu--the
> author of IOLib.
> 
> Here is a link to a very recent edition of the tutorial:
> 
> http://pages.cs.wisc.edu/~psilord/blog/data/iolib-tutorial/tutorial.html
> 

Wow, excellent. Now get over to the "free will" thread and do some real 
work.

I remember back in the day there were three strikes against CL: no 
standard FFI, no standard GUI, and no standard sockets. CFFI covers one 
of those. In your experience/opinion does IOLib answer the sockets 
objection?

kt


-- 
http://www.stuckonalgebra.com
"The best Algebra tutorial program I have seen... in a class by itself." 
Macworld

Kenneth Tilton <kentilton@gmail.com> wrote:
> Wow, excellent.

Thank you.

> Now get over to the "free will" thread and do some real work.

One might say I've exercized my free will and stayed out of that thread. :)

> I remember back in the day there were three strikes against CL: no 
> standard FFI, no standard GUI, and no standard sockets. CFFI covers one 
> of those. In your experience/opinion does IOLib answer the sockets 
> objection?

Yes. I found the API to be easy to use and the socket feature support extensive
enough for my moderate needs. So far my application has scaled to 300 clients
and it looks like I'm not any where near straining IOLib's internals with
respect to network reads and writes.

It is my opinion that if more people used, beat upon, and contributed back to
IOLib, it would be an enterprise class CL networking library in a _very short_
order of time. It needs a port to Windows, which unfortuantely might be
non-trivial, but if it got it then I would reckon IOLib would be the final say
for network programming in CL across most important OSes and architectures.

Thank you.

-pete

 KT> standard FFI, no standard GUI, and no standard sockets. CFFI covers one
 KT> of those. In your experience/opinion does IOLib answer the sockets 
 KT> objection?

What's wrong with usocket?

Captain Obvious <udodenko@users.sourceforge.net> wrote:
> KT> standard FFI, no standard GUI, and no standard sockets. CFFI covers one
> KT> of those. In your experience/opinion does IOLib answer the sockets 
> KT> objection?
> 
> What's wrong with usocket?

In my case, usocket was too much of a thin compatibility layer. I didn't want
to write an i/o multiplexer, which was a requirement, over it. IMHO any serious
codes which utilizes thousands or tens of thousands of clients will need an
event multiplexer to deal with the nonblocking and often asynchronous i/o.  Why
should I write one over usocket when a very good quality one was written
already? This is why I think IOLib is a great library.

Thank you.

-pete

On May 29, 5:30=A0pm, Peter Keller <psil...@cs.wisc.edu> wrote:
> Hello,
>
> I've written an IOLib tutorial which describes how to write network
> clients and servers with IPV4 TCP using blocking I/O with the stream
> interface and nonblocking I/O with the event multiplexer also including
> send-to and receive-from.

Thanks for the well-written and content packed work!

I have a question about ex4:

     (unwind-protect
       (multiple-value-bind (who port)
           (remote-name *ex4-tls-client*)
         (format t "A thread is handling the connection from ~A:~A!~%"
                 who port)

         ;; Prepare the time and send it to the client.
         (multiple-value-bind (s m h d mon y)
             (get-decoded-time)
           (handler-case
               (progn
                 (format t "Sending the time to ~A:~A..." who port)
                 (format *ex4-tls-client*
                         "~A/~A/~A ~A:~A:~A~%"
                         mon d y h m s)
                 (finish-output *ex4-tls-client*)
                 (format t "Sent!~%"))

             (socket-connection-reset-error ()
               (format t "Client ~A:~A reset the connection!~%" who
port))

             (hangup ()
               (format t "Client ~A:~A closed connection.~%" who
port)))))

    ;; Cleanup form for uw-p.
    (format t "Closing connection to ~A:~A!~%"
            (remote-host *ex4-tls-client*) (remote-port *ex4-tls-
client*))
    (close *ex4-tls-client*))

	It is a bit tricky to robustly handle closing of the client socket in
the
	thread. For example, if we bound the special variable *ex4-tls-
client* to a
	lexically scoped variable and then did the UNWIND-PROTECT form to
close the
	lexically scoped variable, then if this thread wakes up and gets
destroyed
	after the lexical binding, but before the UNWIND-PROTECT, we'd lose a
	socket to a client into the garbage collector.

    Such incorrect code would look like:

    ;; This code is incorrect!
    (defun process-ex4-client-thread ()
      (declare (ignorable *ex4-tls-client*))
      (let ((client *ex4-tls-thread*))
        ;; thread gets destroyed right here! client socket is left
open!
        (unwind-protect
          ( <evaluable form=3D""> )
          (close client))))

Aside/unrelated: you forgot to escape some html stuff in that
"incorrect" code example here, it seems.

But my question concerns the difference between lexical vs the
special variable *ex4-tls-client*. Could you (or anyone) elaborate
on this?

szergling <senatorzergling@gmail.com> wrote:
> Thanks for the well-written and content packed work!

Thank you!

>    ;; This code is incorrect!
>    (defun process-ex4-client-thread ()
>      (declare (ignorable *ex4-tls-client*))
>      (let ((client *ex4-tls-thread*))
>        ;; thread gets destroyed right here! client socket is left
> open!
>        (unwind-protect
>          ( <evaluable form=""> )
>          (close client))))


> Aside/unrelated: you forgot to escape some html stuff in that
> "incorrect" code example here, it seems.

In my local web copy of the tutorial, I've fixed this html quoting issue. Thank
you for pointing it out.

> But my question concerns the difference between lexical vs the
> special variable *ex4-tls-client*. Could you (or anyone) elaborate
> on this?

I'll try and clarify the situation.

*ex4-tls-client* is a special variable at all times. When the
process-ex4-client-thread is born, *ex4-tls-client* is bound to whatever socket
was specified in the *default-special-bindings* alist just before make-thread
was called.

If for whatever reason you wanted to rebind the locally bound (to the thread)
special variable into a lexical variable closed to the thread and only operate
on that variable inside of an unwind-protect, there is a race condition in that
if the thread is killed after the binding happens, but before the
unwind-protect is set up, you can leak the socket since only the thread has the
reponsibility to close it.  What I have labeled as incorrect code is an
_explicit race condition_ concerning the mandatory cleanup of the client socket
with respect to unwind-protect. In the incorrect code, there is a distinct
continuation you can point to which has a live socket, but no protection form
to clean it up.

Of course, I'm making all kinds of assumptions, even in what I term "correct"
code. 

The first major assumption that I am making, which I can't really hold to be
true or portable in good faith is:

  The thread cannot be killed between the last instruction needed to create and
  run the thread in make-thread and the sequence of instructions to set up the
  unwind-protect context in process-ex4-client-thread. If in fact this could
  happen, the socket would be lost and not cleaned up.

This is an _implicit race condition_ that the programmer can't control no
matter what they write (to the best of my knowledge) and it seems to be the
best I'm able to do with Bordeaux Threads--or possibly any threading package.
BT's make-thread looks like it either returns a created and alive thread or
signals an error if threading is not supported. It leaves unspecified if a
problem happened during thread creation and the thread could not be created.
One might suppose you could use thread-alive-p to determine if the returned
thread was actually alive, but that function call is not recommended for use in
production code according to the docs, and the thread might have legitimately
come and gone by the time the parent thread was rescheduled, so what to do?

The second major assumption that I am making:

  When a thread is killed for whatever reason, either internally or externally,
  the cleanup form of the unwind-protect is always evaluated.

There is simply no guarantee that this is true. It would appear that
unwind-protect and threading just don't play well together.

Threading in lisp is a thorny issue (and one I'm not well versed in either) and
this is why I left it as minimal as possible with hefty warnings about them in
the examples that use them. I have just enough there to show it can be done,
but to solve all of the threading issues would require more work for the
application writer with support from their specific lisp implementation.

Thank you.

-pete

On May 31, 7:26 pm, Peter Keller <psil...@cs.wisc.edu> wrote:
> szergling <senatorzergl...@gmail.com> wrote:

> >    ;; This code is incorrect!
> >    (defun process-ex4-client-thread ()
> >      (declare (ignorable *ex4-tls-client*))
> >      (let ((client *ex4-tls-thread*))
> >        ;; thread gets destroyed right here! client socket is left open!

Right. The "here" in your comment refers to <here>, as in right
here. I mistakenly assumed that this comment block annotates the
unwind-protect block below, and that the thread is killed/destroyed
inside it...

> >        (unwind-protect
> >          ( <evaluable form=""> )
> >          (close client))))

<<snip>>

> ... there is a race condition in that if the thread is killed after
> the binding happens, but before the unwind-protect is set up, you
> can leak the socket since only the thread has the reponsibility to
> close it.

I see where I was confused now. This is not due to the lexical binding
per se, but could happen with any code like

(progn
  (foo)
  (bar)
  (unwind-protect ...))

The delay before we enter the unwind-protect block leaves us
vulnerable to "leaky" sockets if the current thread is destroyed (or
either foo or bar performs any non-local exits).

More generally, even if there are no apparent high-level expressions
between thread creation and entry into unwind-protect, there could be
many low-level assembler instructions, giving a similar delay. So how
safe is assumption 1 below?

<<snipped full explanations>>

> The first major assumption that I am making, which I can't really
> hold to be true or portable in good faith is:
>
>   The thread cannot be killed between the last instruction needed to
>   create and run the thread in make-thread and the sequence of
>   instructions to set up the unwind-protect context in
>   process-ex4-client-thread. If in fact this could happen, the
>   socket would be lost and not cleaned up.

<<snip>>

> The second major assumption that I am making:
>
>   When a thread is killed for whatever reason, either internally or
>   externally, the cleanup form of the unwind-protect is always
>   evaluated.

How safe is this assumption? Does anyone know any implementations well
enough to comment?

Thank you very much for the explanation, Pete. I continue to learn
about my blind spots with regard to Lisp everyday.

Yong

szergling <senatorzergling@gmail.com> wrote:
> I see where I was confused now. This is not due to the lexical binding
> per se, but could happen with any code like
> 
> (progn
>  (foo)
>  (bar)
>  (unwind-protect ...))

Indeed that is correct.

> The delay before we enter the unwind-protect block leaves us
> vulnerable to "leaky" sockets if the current thread is destroyed (or
> either foo or bar performs any non-local exits).

> More generally, even if there are no apparent high-level expressions
> between thread creation and entry into unwind-protect, there could be
> many low-level assembler instructions, giving a similar delay.

Of course, it is all implementation dependent, but this is true.  Garbage
collection routines or whatnot could be scheduled there and execute in the new
thread, but not in the unwind-protect. Unless one actually went and looked to
see what compilers were actually doing or it was some kind af agreed upon
semantic (which if it is I am unaware of it--but that would be due to simply me
not knowing of it) this is what you have to assume.

> So how safe is assumption 1 below?
> 
>> The first major assumption that I am making, which I can't really
>> hold to be true or portable in good faith is:
>>
>>   The thread cannot be killed between the last instruction needed to
>>   create and run the thread in make-thread and the sequence of
>>   instructions to set up the unwind-protect context in
>>   process-ex4-client-thread. If in fact this could happen, the
>>   socket would be lost and not cleaned up.
> 
>> The second major assumption that I am making:
>>
>>   When a thread is killed for whatever reason, either internally or
>>   externally, the cleanup form of the unwind-protect is always
>>   evaluated.
> 
> How safe is this assumption? Does anyone know any implementations well
> enough to comment?

Unfortunately, I am unable to answer these questions to the depth they deserve
at point in my journey of learning lisp.

Maybe someone else can help out answering them.

Later,
-pete

 KT>>> standard FFI, no standard GUI, and no standard sockets. CFFI covers
 KT>>> one of those. In your experience/opinion does IOLib answer the
 KT>>> sockets objection?
 ??>>
 ??>> What's wrong with usocket?

 PK> In my case, usocket was too much of a thin compatibility layer.

Yes, and being minimal is exactly why it might be good as the standard 
sockets API.
Standard is not supposed to be fancy, it is supposed to be the lowest common 
denominator, I think.

 PK>  I didn't want to write an i/o multiplexer, which was a requirement,
 PK> over it.
 PK>  IMHO any serious codes which utilizes thousands or tens of thousands
 PK> of clients will need an event multiplexer to deal with the nonblocking
 PK> and often asynchronous i/o.  Why should I write one over usocket when a
 PK> very good quality one was written already? This is why I
 PK> think IOLib is a great library.

Yep, sure.

But I believe most applications do not need thousands of simultaneous 
clients and thus do not need i/o multiplexer -- threads should be able to 
cope with up to hundred of connections, I think.
And so I would not rule out usocket. 

Captain Obvious <udodenko@users.sourceforge.net> wrote:
> KT>>> standard FFI, no standard GUI, and no standard sockets. CFFI covers
> KT>>> one of those. In your experience/opinion does IOLib answer the
> KT>>> sockets objection?
> ??>>
> ??>> What's wrong with usocket?
> 
> PK> In my case, usocket was too much of a thin compatibility layer.
> 
> Yes, and being minimal is exactly why it might be good as the standard 
> sockets API.
> Standard is not supposed to be fancy, it is supposed to be the lowest common 
> denominator, I think.

It is only very recently (a week ago?) that I have discovered that IOLIb and
cltl3 have some kind of a relation.

I'm not smart enough to determine if usocket or IOLib should be part of a
standard or not.

> But I believe most applications do not need thousands of simultaneous 
> clients and thus do not need i/o multiplexer -- threads should be able to 
> cope with up to hundred of connections, I think.
> And so I would not rule out usocket. 

Different things for different people, from the usocket website:

"If you want a thin compatibility layer then usocket is the right choice, but
if you want the full socket API then you should take a look at iolib."

Every network application I intend to write in CL will use IOLib, since every
one of them will require lots of buffer management, multiplexed, asynchronous
I/O, and access to the full socket API, and most likely IPV6 in the near
future.

Later,
-pete

Peter Keller wrote:
> Captain Obvious <udodenko@users.sourceforge.net> wrote:
>> KT>>> standard FFI, no standard GUI, and no standard sockets. CFFI covers
>> KT>>> one of those. In your experience/opinion does IOLib answer the
>> KT>>> sockets objection?
>> ??>>
>> ??>> What's wrong with usocket?
>>
>> PK> In my case, usocket was too much of a thin compatibility layer.
>>
>> Yes, and being minimal is exactly why it might be good as the standard 
>> sockets API.
>> Standard is not supposed to be fancy, it is supposed to be the lowest common 
>> denominator, I think.
> 
> It is only very recently (a week ago?) that I have discovered that IOLIb and
> cltl3 have some kind of a relation.
> 
> I'm not smart enough to determine if usocket or IOLib should be part of a
> standard or not.

The spirit of the OQ was, When can Lisp stop apologizing for not having 
the kind of socket support other languages brag about. I would presume 
other languages have solutions closer to IOLib than usocket.

As for the standard, that's an orthogonal concern. Sockets as a de facto 
standard is enough, and for my money does not belong in a language spec.

kt

-- 
http://www.stuckonalgebra.com
"The best Algebra tutorial program I have seen... in a class by itself." 
Macworld