Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
Network Programming in ANSI Common Lisp with IOLib
by: Peter Keller (psilord@cs.wisc.edu)
Version 0.0
4/02/2010
What is IOLib?
--------------
IOLib is a portable I/O library for ANSI Common Lisp. It includes socket
interfaces for network programming with IPV4/IPV6 TCP and UDP, an I/O
multiplexer that includes nonblocking I/O, a DNS resolver library, and a
pathname library.
Where do I get IOLib?
---------------------
The current version of IOLib is found here:
http://common-lisp.net/project/iolib/download.shtml
Please use the repository located in the Live Sources section for the most up
to date version of IOLib.
Introduction
------------
This tutorial loosely follows the exposition of network programming in "UNIX
Network Programming, Networking APIs: Sockets and XTI 2nd Edition" by W.
Richard Stevens. Many examples are derived from the source codes in that book.
Major deviations from the C sources include converting the concurrent examples
which use fork() into threaded examples which use the portable Bordeaux Threads
package, more structured implementations of certain concepts such as data
buffers and error handling, and general movement of coding style towards a
Common Lisp viewpoint.
The scope of this version of the tutorial is:
0. Exposition suitable for programmers unfamiliar with ANSI Common Lisp
1. IPV4 TCP
2. Client/Server architecture
3. Iterative vs Concurrent (via threading) vs Multiplexed Server Design
4. Blocking and nonblocking I/O
It is intended, however, that this tutorial grows to contain the entirety of
IOLib's API as detailed in the Future Directions section of this tutorial. As
newer revisions of this tutorial are released, those gaps will be filled until
the whole of the IOLib API has been discussed.
Finally, the example code in this tutorial is algorithmically cut from the
actual example programs and inserted into the tutorial via a template
generation method. The example codes have embedded in them a tiny markup
language which facilitates this in the form (on a single line) of ';; ex-NNNb'
to begin an example section, and ';; ex-NNNe' to end an example section--NNN
stands for an enumeration integer for which each section's begin and end must
match.
Acknowledgements
----------------
I would like to greatly thank Stelian Ionescu, the author of IOLib
for his exposition of the various features of IOLib and his patience
in our sometimes long conversations.
Supporting Code
---------------
The file package.lisp contains a small library of codes used widely in the
examples. The supporting code implements:
0. The package containing the examples, called :iolib.examples.
1. The variables *host* and *port*, set to "localhost" and 9999
respectively. This is the default name and port to which
client connect and servers listen. Servers usually bind
to 0.0.0.0, however.
2. A small, but efficient, queue implementation, from "ANSI Common Lisp"
by Paul Graham. The interface calls are:
(make-queue)
(enqueue obj q)
(dequeue q)
(empty-queue q)
3. :iolib.examples currently depends upon IOLib alone and uses
packages :common-lisp, :iolib, and :bordeaux-threads.
Running the Examples
--------------------
These examples were developed and tested on SBCL 1.0.33.30 running on an x86
Ubuntu 8.10 machine. They were ran with two sessions of SBCL running, one
acting as a client, and the other as a server.
Supposing we'd like to start up the first example of the daytime server and
connect to it with the first daytime client example. Initially, the server will
bind to *host* and *port* and wait for the client to connect. We connect with
the client to *host* and *port*, get the time, and exit.
First we'll start up a server:
Linux black > sbcl
This is SBCL 1.0.33.30, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* (require :iolib.examples) ; much output!
* (in-package :iolib.examples)
#<PACKAGE "IOLIB.EXAMPLES">
* (run-ex1-server)
Created socket: #<passive IPv4 stream socket, unbound {BF84B99}>[fd=5]
Bound socket: #<passive IPv4 stream socket bound to 0.0.0.0/9999 {BF84B99}>
Listening on socket bound to: 0.0.0.0:9999
Waiting to accept a connection...
[ server is waiting for the below client to connect! ]
Got a connection from 127.0.0.1:34794!
Sending the time...Sent!
T
*
Now we'll start up the client which connected to the above server:
Linux black > sbcl
This is SBCL 1.0.33.30, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* (require :iolib.examples) ; much output!
* (in-package :iolib.examples)
#<PACKAGE "IOLIB.EXAMPLES">
* (run-ex1-client)
Connected to server 127.0.0.1:9999 via my local connection at 127.0.0.1:34794!
2/27/2010 13:51:48
T
*
In each client example, one can specify which host or port to which it should
connect:
* (run-ex1-client :host "localhost" :port 9999)
Connected to server 127.0.0.1:9999 via my local connection at 127.0.0.1:34798!
2/27/2010 13:53:7
T
*
The servers can be told a port they should listen upon and in this tutorial,
unless otherwise specified, will always bind to 0.0.0.0:9999 which means across
all interfaces on the machine and on port 9999.
CHAPTER 1
---------
IPV4 TCP Client/Server
Blocking and nonblocking I/O
Overview of Examples
--------------------
The examples consist of a collection of clients and servers. They are split
into two groups: a set of daytime clients and server, and echo clients and
servers. In some of the examples, a certain network protocol, suppose
end-of-file handling, must be matched between client and server causing further
delineation.
Client protocols are matched to server protocols thusly:
Clients: ex1-client, ex2-client, ex3-client, can work
with servers: ex1-server, ex2-server, ex3-server, ex4-server.
Clients: ex4-client, ex5a-client, can work with servers: ex5-server,
ex6-server.
Clients: ex5b-client, can work with servers: ex7-server, ex8-server
Some clients and servers use the "daytime" series of protocols, those
are ex1-client, ex2-client, ex3-client, and ex1-server, ex2-server,
ex3-server, and ex4-server.
Some clients and servers use the "echo a line" series of protocols,
those are ex4-client, ex5a-client, ex5b-client, and ex5-server,
ex6-server, ex7-server, and ex8-server.
Even though much of the example source is included in the tutorial, it is
recommended that the example sources be carefully read and understood in order
to gain the most benefit from the tutorial.
Daytime Clients
---------------
In this section we show the evolution of a client which connects to a server
and gets the time of day. Each example shows some kind of an incremental
improvement to the previous one.
Daytime Client IVP4/TCP: ex1-client.lisp
----------------------------------------
This example is a very simple daytime client program which contacts a server,
by default at *host* and *port*, returns a single line of text that is the
current date and time, and then exits. It is written in more of a C style just
to make it easy to compare with similar simple examples in other languages. It
uses blocking, line oriented I/O.
The steps this program performs are:
0. The ex1-client.lisp entrance call:
<example ex1-client:ex-0>
1. Create an active TCP socket:
The socket creation function (MAKE-SOCKET ...) is the method by which one
creates a socket in IOLib. It is very versatile and can be used to both
create and initialize the socket in a single call.
In this case, we use it simply and create an active IPV4 Internet stream
socket which can read or write utf8 text and that understands a particular
newline convention in the underlying data.
One small, but important, deviation of IOLib sockets from Berkeley sockets
is that when a socket is created, it is predestined to forever and
unalterably be either an active or passive socket. Active sockets are used
to connect to a server and passive sockets are used for a server's
listening socket.
<example ex1-client:ex-1>
2. Specify the Server's IP address and port and establish a connection
with the server:
This bit of code contains many calls into IOLib and we shall examine each
of them.
The function LOOKUP-HOSTNAME takes as a string the DNS name
for a machine and returns 4 values:
A. an address
B. a list of additional addresses(if existent)
C. the canonical name of the host
D. an alist of all the host's names with their respective addresses
We use only the first return value, the address component, to pass to the
function CONNECT.
The function CONNECT will connect the socket to the address, but to a
random port if the :port keyword argument is not specified. The average
client codes usually use :wait t to block until the connect can resolve
with a connected fd or an error. The exception to always using :wait t is
if the client needs to connect to many servers at once, suppose a web
client, or if a server is also a client in other contexts and wishes not to
block.
The functions REMOTE-HOST and REMOTE-PORT return the ip address and port of
the remote connection associated with the connected socket. LOCAL-HOST and
LOCAL-PORT return the information of the client's end of the connected
socket. Analogous calls REMOTE-NAME and LOCAL-NAME each return two values
where the first value is the equivalent of *-host and the second value is
the equivalent of *-port.
<example ex1-client:ex-2>
3. Read and display the server's reply:
Now that the socket has been connected to the server, the server will send
a line of text to the client. The client uses the standard Common Lisp
function READ-LINE to read the information from the socket. The function
READ-LINE blocks and will only return when an *entire line* is read. Once
read, the line is emitted to *standard-output* via the function call
FORMAT.
<example ex1-client:ex-3>
4. End program:
We close the socket with the standard function CLOSE and return true so the
return value of this example is t.
<example ex1-client:ex-4>
While this program works, it has some major flaws in it. First and foremost is
that it doesn't handle any conditions that IOLib signals in common use cases.
An example would be to run the ex1-client.lisp example without a daytime server
running. In most, if not all, Common Lisp toplevels, you'll be dropped into the
debugger on an unhandled SOCKET-CONNECTION-REFUSED-ERROR condition. Secondly,
it isn't written in the Common Lisp style.
Daytime Client IVP4/TCP: ex2-client.lisp
----------------------------------------
In this example, we simply tackle the fact ex1-server.lisp can be shortened
with an IOLib form to something where the application writer has less to do
concerning cleaning up the socket object. It also uses line oriented blocking
I/O.
The introduced macro WITH-OPEN-SOCKET calls MAKE-SOCKET with the arguments in
question and binds the socket to the variable 'socket'. When this form returns,
it will automatically close the socket.
This shortens the program so much, that the example can be included in its
entirety:
<example ex2-client:ex-0>
This shorthand can go even further, if we add this to the WITH-OPEN-SOCKET
flags
:remote-host (lookup-hostname host)
:remote-port port
then the underlying MAKE-SOCKET call will in fact connect the socket directly
to the server before it is available for the body of the macro allowing us to
remove the connect call entirely! In the early examples, however, we don't
utilize IOLib's shorthand notations to this degree in order to make apparent
how the library maps into traditional socket concepts. After one gains
familiarity with the IOLib API, the situations where application of the
shortcuts are useful become much easier to see.
Daytime Client IVP4/TCP: ex3-client.lisp
----------------------------------------
Now we come to condition handling, which can moderately affect the layout of
your IOLib program. Any real program using IOLib must handle IOLib's signaled
conditions which are common to the boundary cases of network programming.
We've already seen one of these boundary cases when we tried to connect a
daytime client to a server that wasn't running. The condition signaled in that
case was: SOCKET-CONNECTION-REFUSED-ERROR. The stream interface has a set of
conditions which IOLib will signal, and another lower level IOLib layer--which
we'll come to in the nonblocking I/O examples have another set of conditions.
There is some intersection between them and we will explore that later. For
now, we'll just use the conditions associated with a stream.
Our rewrite of ex2-client.lisp into ex3-client.lisp (continuing to use line
oriented blocking I/O) proceeds thusly:
0. We create a helper function which connects to the server and reads the
daytime line:
Notice the HANDLER-CASE macro around the portion of the function which
reads the date from the server. In looking at the boundary conditions from
the server given this protocol, we can receive an END-OF-FILE condition if
the client connected, but before the server could respond it exited,
closing the connection. Since in this case we're inside of a
WITH-OPEN-SOCKET form, we simply note that we got an END-OF-FILE and let
the cleanup forms of WITH-OPEN-SOCKET close the connection. If we don't
catch this condition, then the program will break into the debugger and
that isn't useful. It is usually debatable as to where one should handle
conditions: either near to or far away from the generating calls. In these
simple examples, no choice has any significant pros or cons. As your IOLib
programs become more and more complex, however, it becomes more obvious at
what abstraction level to handle signaled conditions.
<example ex3-client:ex-0>
1. Some conditions which are complete show-stoppers to the functioning of the
code are caught at a higher level:
Notice we catch the possible SOCKET-CONNECTION-REFUSED-ERROR from the
connect inside of the function run-ex3-client-helper.
<example ex3-client:ex-1>
Here are some common conditions in IOLib (some from ANSI Common Lisp too) and
under what situations they are signaled. In any IOLib program, *at least*
these conditions should be handled where appropriate.
END-OF-FILE:
When a stream function such as READ, READ-LINE, etc...(but not
RECEIVE-FROM), reads from a socket where the other end has been closed.
HANGUP:
When writing to a socket with a stream function such as WRITE,
FORMAT, etc...(but not SEND-TO), if the socket is closed then this
condition is signaled.
SOCKET-CONNECTION-RESET-ERROR:
When doing I/O on a socket and the other side of the socket sent a
RST packet, this condition is signaled. It can also happen with
the IOLIb function ACCEPT and similar.
SOCKET-CONNECTION-REFUSED-ERROR:
Signaled by connect if there is no server waiting to accept the incoming
connection.
Daytime Servers
---------------
Now that we have completed the evolution of the daytime client, let's look at
the daytime servers.
The exposition of the servers follows in style of the clients.
Daytime Server IVP4/TCP: ex1-server.lisp
----------------------------------------
This first example is an iterative server which handles a single client and
then exits. The I/O is blocking and no error handling is performed. This is
similar in scope to the ex1-client.lisp example.
0. Create the server socket:
We see that the socket is :passive. Every socket in IOLib is predestined to
be either an active or passive socket and since this is a server socket, it
is passive. Also here we see that we can ask for the underlying fd of the
socket with the function SOCKET-OS-FD.
<example ex1-server:ex-0>
1. Bind the socket
Binding a socket is what gives it an endpoint to which clients can connect.
The IOLib constant +IPV4-UNSPECIFIED+ represents 0.0.0.0 and means if a
connection arrives on any interface, it will be accepted if it comes to the
:port specified. The :reuse-addr keyword represents the socket option
SO_REUSEADDR and states (among other things) that if the socket is in the
TIME_WAIT state it can be reused immediately. It is recommended that all
servers use :reuse-addr on their listening socket.
<example ex1-server:ex-1>
2. Listen on the socket
Listening on a socket allows clients to connect. In this example, we've
specified that 5 pending connection can be queued up in the kernel before
being accepted by the process.
<example ex1-server:ex-2>
3. Accept the client connection.
Here we finally call the IOLib function ACCEPT-CONNECTION. We would like it
to block, so we pass it :wait t. When ACCEPT-CONNECTION returns it will
return a new socket which represents the connection to the client.
ACCEPT-CONNECTION can return nil under some situations, such as on a slow
server when the client sent a TCP RST packet in between the time the kernel
sees the connection attempt and ACCEPT-CONNECTION is actually called. We
also opt to use the function REMOTE-NAME, which returns two values, the ip
address and port of the remote side of the socket.
<example ex1-server:ex-3>
4. Write the time to the client.
Here we've figured out the time string and wrote it to the client. Notice
we call the function FINISH-OUTPUT. This ensures that all output is written
to the client socket. For streams using blocking I/O, it is recommended
that every write to a blocking socket be followed up with a call to
FINISH-OUTPUT.
<example ex1-server:ex-4>
5. Close the connection to the client.
We're done writing to the client, so close the connection so the client
knows it got everything.
<example ex1-server:ex-5>
6. Close the server's socket.
Since this is a one shot server, we close the listening socket and exit. In
this and all other servers we call FINISH-OUTPUT to flush all pending
message to *standard-output*, if any.
<example ex1-server:ex-6>
The above code is the basic idea for how a very simple TCP blocking I/O server
functions. Like ex1-client, this server suffers from the inability to handle
common signaled conditions such as a HANGUP from the client--which means the
client went away before the server could write the time to it.
However, one major, and subtle, problem of this particular example is that the
socket to the client is *not immediately closed* if the server happens to exit,
say by going through the debugger back to toplevel--or a signaled condition,
before writing the date to the client. If this happens, it can take a VERY long
time for the socket to be garbage collected and closed. In this scenario, the
client will hang around waiting for data which will never come until the Lisp
implementation closes the socket when it gets around to collecting it. Garbage
collection is an extremely nice feature of Common Lisp, but non-memory OS
resources in general should be eagerly cleaned up. Clients can suffer from
this problem too, leaving open, but unmanipulable, sockets to servers.
All clients or servers written against IOLib should either use some IOLib
specific macros to handle closing of socket, Common Lisp's condition system
like handler-case to catch the signaled conditions, or some other manual
solution.
Daytime Server IVP4/TCP: ex2-server.lisp
----------------------------------------
Similarly to ex2-client, this server uses the macro WITH-OPEN-SOCKET to open
the server socket. We introduce WITH-ACCEPT-CONNECTION to accept the client and
convert this server from a single shot server to an iterative server which can
handle, in a serial fashion only, multiple clients.
0. Serially accept and process clients:
This portion of ex2-server shows the infinite loop around the accepting of
the connection. The macro WITH-ACCEPT-CONNECTION takes the server socket
and introduces a new binding: client, which is the accepted connection. We
ensure to tell the accept we'd like to be blocking. If for whatever reason
we exit the body, it'll clean up the client socket automatically.
<example ex2-server:ex-0>
For very simple blocking I/O servers like this one, serially accepting and
handling client connections isn't so much of a problem, but if the server does
anything which takes a lot of time or has to send lots of data back and forth
to many persistent clients, then this is a poor design. The means by which you
exit this server is by breaking evaluation and returning to the toplevel. When
this happens, the WITH-* forms automatically close the connection to the
client.
Daytime Server IVP4/TCP: ex3-server.lisp
----------------------------------------
In this iterative and blocking I/O server example, we add the handling of the
usual signaled conditions in network boundary cases often found with sockets.
Like the earlier client where we introduced HANDLER-CASE, this involves a
little bit of restructuring of the codes.
0. A helper function which opens a passive socket, binds it, and
listens on it:
There is nothing new in this portion of the code. We've seen this pattern
before. In production code, we could probably shorten this further by
having WITH-OPEN-SOCKET do the binding and connecting with appropriate
keyword arguments.
<example ex3-server:ex-0>
1. Repeatedly handle clients in a serial fashion:
The new material in this function is the HANDLER-CASE around sending the
client the time information. The boundary conditions when writing to a
client include the server getting a reset (RST) from the client or
discovering the client had gone away and there is no-one to which to write.
Since the write is contained within the WITH-ACCEPT-CONNECTION form, if any
of these conditions happen, we simply notice that they happened and let the
form clean up the socket when it exits. If we didn't catch the conditions,
however, we'd break into the debugger.
One might ask what the value of catching these conditions here is at all
since we don't actually do anything with them--other than printing a
message and preventing the code from breaking into the debugger. For the
purposes of the tutorial, it is intended that the reader induce the
boundary cases manually and see the flow of the code and to understand
exactly what conditions may be signaled under what conditions and how to
structure code to deal with them. In production code where the author might
not care about these conditions at all, one might simply ignore all the
signaled conditions that writing to the client might cause.
Of course, the appropriateness of ignoring network boundary conditions is
best determined by context.
<example ex3-server:ex-1>
2. End of the helper function, returns T to whomever called it:
<example ex3-server:ex-2>
3. The entry point into this example:
We handle the condition SOCKET-ADDRESS-IN-USE-ERROR which is most commonly
signaled when we try to bind a socket to address which already has a server
running on it or when the address is in the TIME_WAIT state. The latter
situation is so common--usually caused by a server just having exited and
another one starting up to replace it, that when binding addresses, one
should supply the keyword argument :reuse-addr with a true value to
BIND-ADDRESS to allow binding a socket to an address in TIME_WAIT state.
<example ex3-server:ex-3>
Daytime Server IVP4/TCP: ex4-server.lisp
----------------------------------------
This is the first of our concurrent servers and the last of our daytime
protocol servers. Usually concurrency is introduced (in the UNIX environment)
with the fork() library call which creates an entirely new process with
copy-on-write semantics to handle the connection to the client. In this
tutorial environment, we've chosen to render this idea with the portable
threading library Bordeaux Threads. The I/O is still line oriented and
blocking, however, when a thread blocks another can run giving the illusion of
a server handling multiple clients in a non-blocking fashion.
We also introduce UNWIND-PROTECT ensures that various sockets are closed under
various boundary conditions in the execution of the server. An UNWIND-PROTECT
executes a single form, and after the evaluation, or interruption, of that
form, evaluates a special cleanup form. The cleanup form is *always* evaluated
and we use this to cleanup non-memory system resources like sockets.
Threads present their own special problems in the design of a server. Two
important problems are: data races and thread termination. The tutorial tries
very hard to avoid any data races in the examples and this problem is
ultimately solvable using Bordeaux-Threads mutexes or condition variables. Our
simple examples do not need mutexes as they do not share any data between
themselves.
The harder problem is thread termination. Since the tutorial encourages
experimentation with the clients and servers in a REPL, threads may leak when
the server process' initial thread stops execution and goes back to the REPL.
We use three API calls from the Bordeaux Threads: THREAD-ALIVE-P, ALL-THREADS,
and DESTROY-THREAD--which are not to be used in normal thread programming. We
do this here in order to try and clean up leaked threads so the clients know
immediately when the server process stopped and we don't pollute the REPL with
an ever increasing number of executing threads. The employed method of
destroying the threads, on SBCL specifically, allows the invocation of the
thread's UNWIND-PROTECT's cleanup form, which closes the socket to the client
before destroying the thread. On other implementations of Common Lisp, we are
not guaranteed that the thread's UNWIND-PROTECT cleanup form will be evaluated
when we destroy it.
This method is also extremely heavy handed in that it uses the function
IGNORE-ERRORS to ignore any condition that Bordeaux Thread's DESTROY-THREAD may
have signaled, including important conditions like HEAP-EXHAUSTED-ERROR, an
SBCL specific condition. In a real threaded server, the exiting of the initial
thread (which means exiting of the runtime and termination of the entire Lisp
process) will destroy all other threads as the process tears itself down and
exits. This is the recommended way a threaded server should exit.
Since threading is implementation dependent for what guarantees are provided,
any non-toy threaded network server will probably use the native implementation
of threads for a specific Common Lisp implementation. An example difficult
situation would be trying to terminate a thread which is blocked on I/O.
Different implementations would handle this in different ways.
The two provided examples, ex4-server and ex5-server, provide a general idea
for the structuring of the code to utilize threads.
Here is the dissection of ex4-server:
0. A special variable which will allow the initial thread to pass a client
socket to a thread handling said client:
<example ex4-server:ex-0>
1. A helper function which begins with the usual recipe for a server:
<example ex4-server:ex-1>
2. Forever more, accept a client connection on the listening socket
and start a thread which handles it:
There is a lot going on in this piece of code. The first thing to notice is
the UNWIND-PROTECT and its cleanup form. The form which UNWIND-PROTECT is
guarding is an infinite loop which does a blocking accept to get a client
socket, rebinds *default-special-bindings* adding to its assoc list the
binding for *ex4-tls-client*, and creates a thread which handles the
client.
The cleanup form walks all of the active client threads and destroys them,
ignoring any conditions that may have arose while doing so. Destroying the
threads prevents them from piling up and eventually causing havoc if many
servers start and exit over time. In addition, it forces an eager close on
the client sockets allowing any clients to know the server went away
immediately.
<example ex4-server:ex-2>
3. The beginning of the thread handling the client:
When the thread is born, the aforementioned explicit binding of the client
socket to *ex4-tls-client* takes effect via the *default-special-bindings*
mechanism. By declaring *ex4-tls-client* ignorable, we inform the compiler
that this variable is set "elsewhere" and no warning should be emitted
about its possibly undefined value. In our case, this will always be
defined at runtime in this server.
<example ex4-server:ex-3>
4. Send the time to the socket:
The UNWIND-PROTECT in this form handles every possible case of leaving the
evaluable function such as it completing normally, a condition being
signaled, or by thread destruction--on SBCL! In all cases, the socket to
the client is closed which cleans up OS resources and lets the client know
right away the server has closed the connection. The HANDLER-CASE form here
just informs us which of the common IOLib conditions may have been signaled
while writing the time to the client.
<example ex4-server:ex-4>
It is a bit tricky to robustly handle closing of the client socket in the
thread. For example, if we bound the special variable *ex4-tls-client* to a
lexically scoped variable and then did the UNWIND-PROTECT form to close the
lexically scoped variable, then if this thread wakes up and gets destroyed
after the lexical binding, but before the UNWIND-PROTECT, we'd lose a
socket to a client into the garbage collector.
Such incorrect code would look like:
;; This code is incorrect!
(defun process-ex4-client-thread ()
(declare (ignorable *ex4-tls-client*))
(let ((client *ex4-tls-thread*))
;; thread gets destroyed right here! client socket is left open!
(unwind-protect
( <evaluable form> )
(close client))))
5. The entry point into this example:
Like earlier servers, we call the helper function and catch what happens if
:reuse-addr wasn't true in the BIND-ADDRESS function call.
<example ex4-server:ex-5>
Daytime Client/Server Commentary
--------------------------------
This concludes the examples using the daytime protocol. We've seen patterns
emerge in how the simplest of clients and servers are built and began to reason
about how to handle common signaled conditions. Threading, of course, increases
the care one must have in order to ensure that data access and control flow is
kept consistent.
Echo Line Clients and Servers
-----------------------------
These next examples focus on the echo protocol. This is simply a server that
sends back to the client whatever the client wrote to it. A client can request
to quit talking to a server (except ex7-server and ex8-server, where this
feature isn't implemented) by sending the word "quit", on a line by itself.
This tells the server to close the connection to the client once it has
finished echoing the line. The closing of the client's read socket lets the
client know the connection to the server went away and that it is time to exit.
We also introduce the socket multiplexer interface which allows concurrent
processing of socket connections. This is similar to how UNIX's select(),
epoll(), or kqueue() works. Due to portability concerns on doing nonblocking
operations on *standard-input* and *standard-output* (we can't easily do it) we
are beholden to some form of blocking I/O in our clients because they interact
with a human. We will explore true non-blocking I/O in the ex8-server example
since that server only has to converse with connected clients.
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Echo Clients
------------
The echo clients are a group of programs which read a line from
*standard-input*, write it to the server, read back the response from the
server, and emit the result to *standard-output*. While there is a portable
method to read "however much is available" from *standard-input*, there isn't
the symmetrical method to write "whatever I'm able" to *standard-output*. For
our client design, this means that all of these clients are line oriented and
do blocking I/O when reading from *standard-input* and writing to
*standard-output*.
Echo Client IPV4/TCP: ex4-client.lisp
--------------------------------------
This is a very basic echo client program that handles the usual conditions
while talking to the server:
0. Connect to the server and start echoing lines:
Here we use WITH-OPEN-SOCKET to create an active socket that we then use to
connect to the server. We handle HANGUP, for when the server went away
before the client could write to it, and END-OF-FILE, for when the server
closes down the connection.
Notice we call the function ex4-str-cli inside of a HANDLER-CASE macro.
This allows us to not check for any signaled conditions in ex4-str-cli and
greatly simplifies its implementation.
In this specific example, we don't do anything other than notify that the
condition happened since after that the socket gets closed via
WITH-OPEN-SOCKET.
<example ex4-client:ex-0>
1. Echo lines to the server:
Until the user inputs "quit" on a line by itself, we read a line, send it
to the server, read it back, and emit it to stdout. If any of the usual
conditions are signaled here, the handler-case in the Step 0 code fires and
we deal with it there.
When "quit" is entered, the line is sent on the round trip to the server
like usual, but this time the server closes the connection to the client.
Unfortunately, since the client is doing blocking I/O, we must read another
line from *standard-input* before we get any signaled condition when IOLib
discovers the socket has been closed by the server.
In practice, this means after the server closed the connection, the user
must hit <return> in order to drive the I/O loop enough to get the signaled
condition.
<example ex4-client:ex-1>
2. Entry point into the example:
We handle the usual connection refused condition, but otherwise this step
is unremarkable.
<example ex4-client:ex-2>
Echo Client IPV4/TCP: ex5a-client.lisp
--------------------------------------
This is the first client to use the socket multiplexer to notice when the
socket to the server is ready for reading or writing. While the multiplexer is
often used in single threaded servers it can be used for clients--especially
clients which may talk to multiple servers like web clients. Use of the
multiplexer API will require a significant change in how the code is
structured. It is not recommended that the multiplexer and threads be used
simultaneously to handle network connections.
Keeping in mind the fact that we ALWAYS could block while reading from
*standard-input* or writing to *standard-output*, we only attempt to read/write
to the standard streams when the multiplexer thinks it can read/write to the
server without blocking. This is a change from the traditional examples of how
to do this in C because in C one can determine if STDIN or STDOUT are ready in
the same manner as a network file descriptor.
The first big change from our previous examples is that we stop using
WITH-OPEN-SOCKET since now we must manually control when the socket to the
server must be closed. This is especially important for clients who use active
sockets. The second change is how we do the creation and registering of the
handlers for reading and writing to the server socket. The third change is how
to unregister a handler and close the socket associated with it under the right
conditions. Other changes will be explained as we meet them.
The main functions of the multiplexer API are:
(make-instance 'iomux:event-base ....)
Create an instance of the event-base, and associate some properties
with it, such as event-dispatch should return if the multiplexer
does not have any sockets it is managing.
Passed an:
:exit-when-empty - when no handlers are registered, event-dispatch
will return.
(event-dispatch ...)
By default, sit in the multiplexer loop forever and handle I/O
requests. It is passed the event-base binding and in addition:
:once-only - run the ready handlers once then return.
:timeout - when there is no I/O for a certain amount of time return.
(set-io-handler ...)
Associates a handler with a state to be called with a specific socket.
Passed an:
event-base binding
:read or :write or :error keyword
the handler closure
(remove-fd-handlers ...)
Removes a handler for a specific state with a specific socket.
Passed an:
event-base binding
an fd
one or more of :read t, :write t, :error t
Here is the example using this API.
0. The event base:
The event-base is the object which holds the state of the multiplexer. It
must be initialized and torn down as we'll see in the entry function to
this example.
<example ex5a-client:ex-0>
1. A helper function in which we create the active socket:
Instead of using WITH-OPEN-SOCKET, we manually create the socket. We do
this to better control how to close the socket. WITH-OPEN-SOCKET will try
to FINISH-OUTPUT on the socket before closing it. This is bad if the socket
had been previously closed or signaled a condition like HANGUP. Trying to
write more data to an already hung up socket will simply signal another
condition. To prevent layers of condition handling code, we explicitly
handle closing of the socket ourselves.
<example ex5a-client:ex-1>
2. Connect to the server, register the socket handlers:
We protect the closing of the socket via UNWIND-PROTECT. We will talk about
the ramifications of this decision in the next step which describes the
UNWIND-PROTECT's cleanup form. In this section of code, we set up a read
and write handler for the socket, and invoke the dispatch function, which
will continue calling the handlers associated with the socket until the
socket gets closed and the handlers unregistered. When this happens (see
the entrance function step for why), EVENT-DISPATCH returns and we continue
on to the cleanup form for the UNWIND-PROTECT.
Setting up a handler in the multiplexer requires several arguments to
the function set-io-handler. Here are what the arguments to that function
are:
1. *ex5a-event-base*
This is the instance of the multiplexer for which we are setting
up the handler.
2. (socket-os-fd socket)
This call returns the underlying operating system's file
descriptor associated with the socket.
3. :read
This keyword states that we'd like to call the handler when the
socket is ready to read. There is also :write and :error.
4. (make-ex5a-str-cli-read socket
(make-ex5a-client-disconnector socket))
The make-ex5a-str-cli-read function returns a closure over the
socket and another closure returned by the
make-ex5a-client-disconnector function. This function is what will
be called when the socket is ready for reading. We will shortly
explain the signature of this function and what gets passed to it
by the multiplexer. The disconnector function will be called by the
returned reader function if the reader function thinks that it
needs to close the socket to the server.
<example ex5a-client:ex-2>
3. Cleanup form for UNWIND-PROTECT:
In the cleanup form, we always close the socket and we pass the function
close :abort t to try and close the socket in any way possible. If we just
tried closing the socket, then we might cause another condition to be
signaled if a previous condition, like HANGUP, had already affected the
socket. :abort t avoids that case. If the socket is already closed by a
handler by the time we get here, closing it again hurts nothing.
<example ex5a-client:ex-3>
4. Make the writer function for when the socket is ready to write:
This function returns a closure which is called by the multiplexer when it
is ready to read something from the server. The arguments to the closure
are fd, the underlying file descriptor for the ready socket, event, which
could be :read, :write, or :error if the handler was registered multiple
times, and exception, which is nil under normal conditions, :error under an
error with the socket, or :timeout, if we were using timeout operations
when dealing with the socket.
The closure will read a line with the function READ-LINE and write it to
the server. The read will be blocking, but hopefully the write won't be
since the multiplexer told us we could perform the write and not block.
Obviously, is we write an enormous line, then we might block again, and in
this case the FINISH-OUTPUT on the socket will push the data in a blocking
I/O fashion until it is done and we return from the handler. So while this
closure for the most part writes when ready, there are cases under which
it'll still block.
In this handler, if there is a signaled condition either reading from
*standard-input* (the END-OF-FILE condition) or writing to the server
socket (the HANGUP condition), we invoke the disconnector closure and pass
it :close. When we get to the description of the disconnector function,
you'll see what that means.
Once the disconnector closure is invoked, the handler will have been
removed and the socket closed. This will make EVENT-DISPATCH return since
the only socket it was multiplexing for was closed--because we've told the
multiplexer to do so when it was made!
<example ex5a-client:ex-4>
5. Make the reader function for when the socket is ready to read:
This piece of code is very similar to the previous step's code, we just
handle the appropriate conditions and after reading the line from the
server emit it to *standard-output*. Again, even though we are told we can
read from the server without blocking, if the read is large enough we will
continue to block until read-line reads the all the data and the newline.
<example ex5a-client:ex-5>
6. The disconnector function:
This function returns a closure which takes an arbitrary number of
arguments. If the arguments to the invoked closure contain :read, :write,
or :error, the respective handler on the associated socket is removed. If
none of those three are supplied, then all handlers for that socket are
removed. Additionally if :close is specified, the socket is closed. While
not all features of this function is used in this example, this function
(or a similar one using the correct event-base special variable) is used
whenever we use the multiplexer in an example.
The closure is called whenever a handler believes it should unregister
itself or another handler, or close the socket. Because we will often close
the socket in the disconnector closure, we can't use WITH-OPEN-SOCKET to
automatically close the socket because WITH-OPEN-SOCKET may try to flush
data on the socket, signaling another condition.
<example ex5a-client:ex-6>
7. The entry point for this example and setting up the event-base:
This function is much more complex than in examples that do not use the
multiplexer. Protected by an UNWIND-PROTECT, we first initialize the event
base my calling make-instance 'iomux:event-base. Here is where we pass the
keyword argument :exit-when-empty t which states that the event-dispatch
function should return when there are no more registered handlers. Once
that is done, we call the helper, catching a common condition and waiting
until we return.
<example ex5a-client:ex-7>
8. The cleanup form for UNWIND-PROTECT: