[version 0.7.0 ediware**20070408050011] { hunk ./CHANGELOG.txt 1 +Version 0.7.0 +2007-04-07 +Switched from trivial-sockets to usocket (patch by Erik Huelsmann) + hunk ./doc/index.html 1 - - - - - - DRAKMA - A Common Lisp web client - - - - - -

DRAKMA - A Common Lisp web client

- -
-
 

Abstract

- -Drakma is a fully-featured web client (implemented in Common Lisp) -that knows how to handle HTTP/1.1 -chunking, persistent -connections, re-usable -sockets, SSL, continuable -uploads, file uploads, cookies, and other -things. And it's probably a result of -my NIH -syndrome... -

-Drakma was developed and tested -with LispWorks, but it should -also work with a couple of other Common Lisp implementations depending -on the supporting libraries. Some tests -with SBCL seem to confirm this. -

-The code comes with -a BSD-style -license so you can basically do with it whatever you want. - -

-Download shortcut: http://weitz.de/files/drakma.tar.gz. -

- -
 

Contents

-
    -
  1. Examples -
  2. Download and installation -
  3. Support and mailing lists -
  4. The Drakma dictionary -
      -
    1. The request -
        -
      1. http-request -
      2. *drakma-default-external-format* -
      3. *header-stream* -
      -
    2. Cookies -
        -
      1. cookie -
      2. cookie-name -
      3. cookie-value -
      4. cookie-domain -
      5. cookie-path -
      6. cookie-expires -
      7. cookie-securep -
      8. cookie-jar -
      9. cookie-jar-cookies -
      10. cookie= -
      11. delete-old-cookies -
      -
    3. Headers -
        -
      1. header-value -
      2. split-tokens -
      3. read-tokens-and-parameters -
      4. parameter-present-p -
      5. parameter-value -
      6. get-content-type -
      -
    -
  5. Acknowledgements -
- -
 

Examples

- -Here's an example session with Drakma which demonstrates some of its -features. (Some linebreaks were added or removed to enhance -legibility.) Note that this doesn't necessarily reflect the current -version of Drakma. The examples should work nevertheless. - -
-;; create a log file of this sessions
-CL-USER 1 > (dribble "/tmp/drakma_dribble")
-; Loading C:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\ccl\dribble.ofasl on demand...
-
-;; load Drakma
-CL-USER 2 > (asdf:oos 'asdf:load-op :drakma)
-; loading system definition from c:\home\lisp\drakma\drakma.asd into
-; #<The ASDF0 package, 0/16 internal, 0/16 external>
-; Loading text file c:\home\lisp\drakma\drakma.asd
-; registering #<SYSTEM :DRAKMA 21D6D24F> as DRAKMA
-;; Creating system COMMON-LISP-USER::DRAKMA
-; loading system definition from c:\home\lisp\chunga\chunga.asd into
-; #<The ASDF0 package, 0/16 internal, 0/16 external>
-; Loading text file c:\home\lisp\chunga\chunga.asd
-; registering #<SYSTEM :CHUNGA 200B12A3> as CHUNGA
-;; Creating system COMMON-LISP-USER::CHUNGA
-; loading system definition from c:\home\lisp\flexi-streams\flexi-streams.asd into
-; #<The ASDF0 package, 0/16 internal, 0/16 external>
-; Loading text file c:\home\lisp\flexi-streams\flexi-streams.asd
-; registering #<SYSTEM :FLEXI-STREAMS 200E8017> as FLEXI-STREAMS
-;; Creating system COMMON-LISP-USER::FLEXI-STREAMS
-; loading system definition from c:\home\lisp\trivial-gray-streams\trivial-gray-streams.asd into
-; #<The ASDF0 package, 0/16 internal, 0/16 external>
-; Loading text file c:\home\lisp\trivial-gray-streams\trivial-gray-streams.asd
-; registering #<SYSTEM :TRIVIAL-GRAY-STREAMS 21D6741F> as TRIVIAL-GRAY-STREAMS
-;; Creating system COMMON-LISP-USER::TRIVIAL-GRAY-STREAMS
-; loading system definition from c:\home\lisp\cl-base64-3.3.2\cl-base64.asd into
-; #<The ASDF0 package, 0/16 internal, 0/16 external>
-; Loading text file c:\home\lisp\cl-base64-3.3.2\cl-base64.asd
-; registering #<SYSTEM CL-BASE64 21D6D277> as CL-BASE64
-;; Creating system COMMON-LISP-USER::CL-BASE64
-; registering #<SYSTEM CL-BASE64-TESTS 2009701B> as CL-BASE64-TESTS
-;; Creating system COMMON-LISP-USER::CL-BASE64-TESTS
-; loading system definition from c:\home\lisp\puri-1.5\puri.asd into
-; #<The ASDF0 package, 0/16 internal, 0/16 external>
-; Loading text file c:\home\lisp\puri-1.5\puri.asd
-; registering #<SYSTEM PURI 21D6B093> as PURI
-;; Creating system COMMON-LISP-USER::PURI
-; registering #<SYSTEM PURI-TESTS 200CFEEF> as PURI-TESTS
-;; Creating system COMMON-LISP-USER::PURI-TESTS
-; Loading fasl file c:\home\lisp\trivial-gray-streams\package.ofasl
-; Loading fasl file c:\home\lisp\trivial-gray-streams\mixin.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\packages.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\ascii.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\iso-8859.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\code-pages.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\specials.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\util.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\external-format.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\in-memory.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\stream.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\output.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\input.ofasl
-; Loading fasl file c:\home\lisp\flexi-streams\strings.ofasl
-; Loading fasl file c:\home\lisp\chunga\packages.ofasl
-; Loading fasl file c:\home\lisp\chunga\specials.ofasl
-; Loading fasl file c:\home\lisp\chunga\util.ofasl
-; Loading fasl file c:\home\lisp\chunga\read.ofasl
-; Loading fasl file c:\home\lisp\chunga\streams.ofasl
-; Loading fasl file c:\home\lisp\chunga\input.ofasl
-; Loading fasl file c:\home\lisp\chunga\output.ofasl
-; Loading fasl file c:\home\lisp\cl-base64-3.3.2\package.ofasl
-; Loading fasl file c:\home\lisp\cl-base64-3.3.2\encode.ofasl
-; Loading fasl file c:\home\lisp\cl-base64-3.3.2\decode.ofasl
-; Loading fasl file c:\home\lisp\puri-1.5\src.ofasl
-; Loading fasl file c:\home\lisp\drakma\packages.ofasl
-; Loading fasl file c:\home\lisp\drakma\specials.ofasl
-; Loading fasl file c:\home\lisp\drakma\util.ofasl
-; Loading c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\comm-defsys.lisp on demand...
-;; Creating system COMM
-
-;  Loading text file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\comm-pkg.lisp
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\sockets.ofasl
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl-constants.ofasl
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl-foreign-types.ofasl
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl.ofasl
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl-certs.ofasl
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0001.ofasl
-; Loaded public patch COMM 1.1
-
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0002.ofasl
-; Loaded public patch COMM 1.2
-
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0003.ofasl
-; Loaded public patch COMM 1.3
-
-;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0004.ofasl
-; Loaded public patch COMM 1.4
-
-; Loading fasl file c:\home\lisp\drakma\read.ofasl
-; Loading fasl file c:\home\lisp\drakma\cookies.ofasl
-; Loading fasl file c:\home\lisp\drakma\request.ofasl
-NIL
-
-;; create a package to work in
-CL-USER 3 > (defpackage :drakma-user (:use :cl :drakma))
-#<The DRAKMA-USER package, 0/16 internal, 0/16 external>
-
-;; switch to this package
-CL-USER 4 > (in-package :drakma-user)
-#<The DRAKMA-USER package, 0/16 internal, 0/16 external>
-
-;; log headers, so we can see what happens -
-;; output to *HEADER-STREAM* will be shown in green below
-DRAKMA-USER 5 > (setq *header-stream* *standard-output*)
-#<Broadcast stream to (#<Echo Stream Input = #<EDITOR::RUBBER-STREAM #<EDITOR:BUFFER CAPI interactive-pane 2> 2198ECD7>,
-                                     Output = #<STREAM::LATIN-1-FILE-STREAM c:\tmp\drakma_dribble>>
-                       #<EDITOR::RUBBER-STREAM #<EDITOR:BUFFER CAPI interactive-pane 2> 2198ECD7>)>
-
-;; note how Drakma automatically follows the 301 redirect and how the fourth return value shows the new URI
-DRAKMA-USER 6 > (http-request "http://lisp.org/")
-GET / HTTP/1.1
-Host: lisp.org
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 301  Moved Permanently
-Date: Sat, 26 Aug 2006 15:46:31 GMT
-Connection: Close
-Server: AllegroServe/1.2.37
-Transfer-Encoding: chunked
-LOCATION: /index.html
-
-GET /index.html HTTP/1.1
-Host: lisp.org
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 200  OK
-Date: Sat, 26 Aug 2006 15:46:32 GMT
-Connection: Close
-Server: AllegroServe/1.2.37
-Content-Type: text/html
-Content-Length: 82
-LAST-MODIFIED: Mon, 16 Feb 2004 09:30:02 GMT
-
-"<title>redirect...</title>
-<meta http-equiv=\"Refresh\" content=\"0; url=/alu/home\">
-"
-200
-((:DATE . "Sat, 26 Aug 2006 15:46:32 GMT")
- (:CONNECTION . "Close")
- (:SERVER . "AllegroServe/1.2.37")
- (:CONTENT-TYPE . "text/html")
- (:CONTENT-LENGTH . "82")
- (:LAST-MODIFIED . "Mon, 16 Feb 2004 09:30:02 GMT"))
-#<URI http://lisp.org/index.html>
-#<FLEXI-STREAMS:FLEXI-IO-STREAM 201017D3>
-T
-
-;; here, Drakma automatically interprets the 'charset=utf-8' part correctly -
-;; might look a bit different in your listener depending on the font you've chosen
-DRAKMA-USER 7 > (subseq (http-request "http://www.cl.cam.ac.uk/~mgk25/ucs/examples/digraphs.txt") 0 298)
-GET /~mgk25/ucs/examples/digraphs.txt HTTP/1.1
-Host: www.cl.cam.ac.uk
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 16:02:56 GMT
-Server: Apache/1.3.37 (Unix) mod_ucam_webauth/1.2.2
-Last-Modified: Thu, 05 Jan 2006 20:49:55 GMT
-ETag: "17cd62-298-43bd8673"
-Accept-Ranges: bytes
-Content-Length: 664
-Connection: close
-Content-Type: text/plain; charset=utf-8
-
-"Latin Digraphs and Ligatures in ISO10646-1
-
-A short table of ligatures and digraphs follows. Some of these may not be
-ligatures/digraphs in the technical sense, (for example, æ is a seperate
-letter in English), but visually they behave that way.
-
-AÆE : U+00C6
-aæe : U+00E6
-ſßs : U+00DF
-IIJJ : U+0132"
-
-;; a vector of octets is returned for (non-text) binary data - a picture in this case
-DRAKMA-USER 8 > (http-request "http://zappa.com/favicon.ico")
-GET /favicon.ico HTTP/1.1
-Host: zappa.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 16:02:59 GMT
-Server: Apache/2.0.46 (Red Hat)
-Last-Modified: Fri, 17 Mar 2006 08:11:07 GMT
-ETag: "3a4080-b6-59d5bcc0"
-Accept-Ranges: bytes
-Content-Length: 182
-Connection: close
-Content-Type: image/gif
-
-#(71 73 70 56 57 97 17 0 17 0 179 1 0 150 151 153 255 255 255 37 37 36 112 114 115
-  201 202 204 0 0 0 80 83 84 26 28 26 230 231 231 249 249 249 12 13 14 219 221 222
-  18 21 22 239 240 241 52 52 54 64 66 66 33 249 4 1 0 0 1 0 44 0 0 0 0 17 0 17 0 0
-  4 99 48 200 73 107 109 54 172 101 129 120 196 180 12 12 51 80 64 161 42 3 48 28
-  170 106 72 141 16 223 120 113 166 121 95 0 14 95 239 33 236 41 98 10 129 114 185
-  188 29 127 25 201 224 73 60 4 8 0 130 22 59 64 52 96 135 148 35 96 80 152 159 186
-  192 64 183 112 0 200 61 65 0 1 192 76 214 185 113 102 241 88 26 90 8 81 18 8 94 
-  130 134 22 17 0 59)
-200
-((:DATE . "Sat, 26 Aug 2006 16:02:59 GMT")
- (:SERVER . "Apache/2.0.46 (Red Hat)")
- (:LAST-MODIFIED . "Fri, 17 Mar 2006 08:11:07 GMT")
- (:ETAG . "\"3a4080-b6-59d5bcc0\"")
- (:ACCEPT-RANGES . "bytes")
- (:CONTENT-LENGTH . "182")
- (:CONNECTION . "close")
- (:CONTENT-TYPE . "image/gif"))
-#<URI http://zappa.com/favicon.ico>
-#<FLEXI-STREAMS:FLEXI-IO-STREAM 200D59BF>
-T
-
-;; a secure connection (see below) -
-;; also note that the server uses chunked transfer encoding for its reply
-DRAKMA-USER 9 > (ppcre:scan-to-strings "(?s)You have.*your data."
-                                       (http-request "https://www.fortify.net/cgi/ssl_2.pl"))
-GET /cgi/ssl_2.pl HTTP/1.1
-Host: www.fortify.net
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 16:10:06 GMT
-Server: Apache
-Connection: close
-Transfer-Encoding: chunked
-Content-Type: text/html
-
-
-"You have connected to this web server using the DHE-RSA-AES256-SHA encryption cipher
- with a key length of 256 bits.
- <p>
- This is a high-grade encryption connection, regarded by most experts as being suitable
- for sending or receiving even the most sensitive or valuable information
- across a network.
- <p>
- In a crude analogy, using this cipher is similar to sending or storing your data inside
- a high quality safe - compared to an export-grade cipher which is similar to using
- a paper envelope to protect your data."
-#()
-
-;; using a different 'User-Agent' header
-DRAKMA-USER 10 > (ppcre:regex-replace-all
-                  "<.*?>"
-                  (ppcre:scan-to-strings "(?s)Your browser reports.*?</table>" 
-                                         (http-request "http://bcheck.scanit.be/bcheck/"
-                                                       :user-agent :explorer))
-                  "")
-GET /bcheck/ HTTP/1.1
-Host: bcheck.scanit.be
-User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
-Accept: */*
-Connection: close
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 16:21:50 GMT
-Server: Apache
-Connection: close
-Transfer-Encoding: chunked
-Content-Type: text/html
-
-
-"Your browser reports to be:
-
-Browser name: MSIE
-Version: 6.0
-Platform: Windows NT 5.1
-"
-
-;; sending parameters in a POST request and working with cookies -
-;; note how Drakma sends the cookie back in the second request
-DRAKMA-USER 11 > (let ((cookie-jar (make-instance 'cookie-jar)))
-                   (http-request "http://www.phpsecurepages.com/test/test.php"
-                                 :method :post
-                                 :parameters '(("entered_login" . "test")
-                                               ("entered_password" . "test"))
-                                 :cookie-jar cookie-jar)
-                   (http-request "http://www.phpsecurepages.com/test/test2.php"
-                                 :cookie-jar cookie-jar)
-                   (cookie-jar-cookies cookie-jar))
-POST /test/test.php HTTP/1.1
-Host: www.phpsecurepages.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Content-Length: 40
-Content-Type: application/x-www-form-urlencoded
-Connection: close
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 18:26:17 GMT
-Server: Apache/2.0.51 (Fedora)
-X-Powered-By: PHP/4.3.10
-Expires: Thu, 19 Nov 1981 08:52:00 GMT
-Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
-Pragma: no-cache
-Set-Cookie: PHPSESSID=3ce33aa3e326ab4bf5da7feecc3248b4; path=/
-Connection: close
-Transfer-Encoding: chunked
-Content-Type: text/html
-
-
-GET /test/test2.php HTTP/1.1
-Host: www.phpsecurepages.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Cookie: PHPSESSID=3ce33aa3e326ab4bf5da7feecc3248b4
-Connection: close
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 18:26:18 GMT
-Server: Apache/2.0.51 (Fedora)
-X-Powered-By: PHP/4.3.10
-Expires: Thu, 19 Nov 1981 08:52:00 GMT
-Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
-Pragma: no-cache
-Connection: close
-Transfer-Encoding: chunked
-Content-Type: text/html
-
-
-(#<COOKIE PHPSESSID=3ce33aa3e326ab4bf5da7feecc3248b4; path=/; domain=www.phpsecurepages.com>)
-
-;; now we are going to re-use a socket for the second connection to the same server
-;; this will also work with chunked encoding
-DRAKMA-USER 12 > (let ((stream (nth-value 4 (http-request "http://www.lispworks.com/" :close nil))))
-                   (nth-value 2 (http-request "http://www.lispworks.com/success-stories/index.html"
-                                              :stream stream)))
-GET / HTTP/1.1
-Host: www.lispworks.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 18:34:20 GMT
-Server: Apache/1.3.37 Ben-SSL/1.57 (Unix)
-Last-Modified: Tue, 08 Aug 2006 18:20:49 GMT
-ETag: "28ee4f0-22db-44d8d601"
-Accept-Ranges: bytes
-Content-Length: 8923
-Content-Type: text/html
-
-GET /success-stories/index.html HTTP/1.1
-Host: www.lispworks.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 200 OK
-Date: Sat, 26 Aug 2006 18:34:20 GMT
-Server: Apache/1.3.37 Ben-SSL/1.57 (Unix)
-Last-Modified: Tue, 08 Aug 2006 18:22:19 GMT
-ETag: "28f3f42-2325-44d8d65b"
-Accept-Ranges: bytes
-Content-Length: 8997
-Connection: close
-Content-Type: text/html
-
-((:DATE . "Sat, 26 Aug 2006 18:34:20 GMT")
- (:SERVER . "Apache/1.3.37 Ben-SSL/1.57 (Unix)")
- (:LAST-MODIFIED . "Tue, 08 Aug 2006 18:22:19 GMT")
- (:ETAG . "\"28f3f42-2325-44d8d65b\"")
- (:ACCEPT-RANGES . "bytes")
- (:CONTENT-LENGTH . "8997")
- (:CONNECTION . "close")
- (:CONTENT-TYPE . "text/html"))
-
-;; testing basic authorization against a local Hunchentoot server
-DRAKMA-USER 13 > (nth-value 1 (http-request "http://localhost:9000/tbnl/test/authorization.html"))
-GET /tbnl/test/authorization.html HTTP/1.1
-Host: localhost:9000
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 401 Authorization Required
-Content-Length: 563
-Content-Type: text/html; charset=iso-8859-1
-Date: Sat, 26 Aug 2006 18:38:58 GMT
-Server: Hunchentoot 0.1.5 (TBNL 0.10.0)
-Connection: Close
-WWW-Authenticate: Basic realm="TBNL"
-
-401
-
-DRAKMA-USER 14 > (nth-value 1 (http-request "http://localhost:9000/tbnl/test/authorization.html"
-                                            :basic-authorization '("nanook" "igloo")))
-GET /tbnl/test/authorization.html HTTP/1.1
-Host: localhost:9000
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Authorization: Basic bmFub29rOmlnbG9v
-Accept: */*
-Connection: close
-
-HTTP/1.1 200 OK
-Content-Length: 884
-Content-Type: text/html; charset=iso-8859-1
-Date: Sat, 26 Aug 2006 18:39:19 GMT
-Server: Hunchentoot 0.1.5 (TBNL 0.10.0)
-Connection: Close
-
-200
-
-;; now we ask Drakma to return a stream and read from it directly
-DRAKMA-USER 15 > (let ((stream (http-request "http://www.jalat.com/blogs/lisp?id=3" 
-                                             :want-stream t)))
-                   (loop for i below 41
-                         for line = (read-line stream)
-                         when (> i 35)
-                         do (write-line line))
-                   (close stream)
-                   (values))
-GET /blogs/lisp?id=3 HTTP/1.1
-Host: www.jalat.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-
-HTTP/1.1 200 OK
-Content-Length: 21453
-Content-Type: text/html; charset=iso-8859-1
-Date: Sat, 26 Aug 2006 19:53:37 GMT
-Server: Hunchentoot 0.1.3 (TBNL 0.9.7)
-Connection: Close
-
-Bill Clementson has <a
-href="http://bc.tech.coop/blog/041111.html">written</a> about getting
-TBNL up and running with apache and mod_lisp. In this example I'm
-going to use <a href="http://weitz.de/hunchentoot/">hunchentoot</a>, a
-pure lisp web server by (again) Edi Weitz.
-
-;; let's test a POST request without content length and with chunked transfer encoding -
-;; we build the content in several steps using different types of data
-;; (note: doesn't work anymore, probably due to server changes)
-DRAKMA-USER 16 > (let ((temp-file (ensure-directories-exist #p"/tmp/quux.txt"))
-                       (continuation (http-request "http://meme.b9.com/login.html"
-                                                   :method :post
-                                                   :content :continuation)))
-                   (funcall continuation "username=" t)
-                   (funcall continuation (list (char-code #\n) (char-code #\a)) t)
-                   (funcall continuation (lambda (stream)
-                                           (write-char #\n stream)) t)
-                   (with-open-file (out temp-file
-                                        :direction :output
-                                        :if-does-not-exist :create
-                                        :if-exists :supersede)
-                     (write-string "ook" out))
-                   (funcall continuation temp-file t)
-                   (ppcre:scan-to-strings "(?i)[a-z ]+nanook[a-z .]+"
-                                          (funcall continuation "&password=igloo")))
-POST /login.html HTTP/1.1
-Host: meme.b9.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-Content-Type: application/x-www-form-urlencoded
-Transfer-Encoding: chunked
-
-HTTP/1.0 200 OK
-Date: Sat, 02 Sep 2006 00:25:24 GMT
-Connection: close
-Server: AllegroServe/1.2.45
-Content-Type: text/html
-Content-Length: 2922
-PRAGMA: no-cache
-CACHE-CONTROL: no-cache
-SET-COOKIE: meme=1834b91d26f9be983a0ed9ca; path=/
-
-"The username nanook is not in our database."
-#()
-
-;; finally, we send additional headers to ask for a range
-DRAKMA-USER 17 > (ppcre:regex-replace-all
-                  "<.*?>"
-                  (format nil "~A~A"
-                          (http-request "http://users.cableaz.com/~lantz/pages/hunchentoot.html"
-                                        :additional-headers '(("Range" . "bytes=959-999")))
-                          (http-request "http://users.cableaz.com/~lantz/pages/hunchentoot.html"
-                                        :additional-headers '(("Range" . "bytes=1165-1201"))))
-                  "")
-GET /~lantz/pages/hunchentoot.html HTTP/1.1
-Host: users.cableaz.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-Range: bytes=959-999
-
-HTTP/1.1 206 Partial Content
-Date: Sat, 26 Aug 2006 19:07:44 GMT
-Server: Apache/2.0.16 (Unix)
-Last-Modified: Sun, 24 Apr 2005 04:08:45 GMT
-ETag: "35298d-2fea-d8f4cd40"
-Accept-Ranges: bytes
-Content-Length: 41
-Content-Range: bytes 959-999/12266
-Content-Type: text/html; charset=ISO-8859-1
-Connection: close
-
-GET /~lantz/pages/hunchentoot.html HTTP/1.1
-Host: users.cableaz.com
-User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
-Accept: */*
-Connection: close
-Range: bytes=1165-1201
-
-HTTP/1.1 206 Partial Content
-Date: Sat, 26 Aug 2006 19:07:45 GMT
-Server: Apache/2.0.16 (Unix)
-Last-Modified: Sun, 24 Apr 2005 04:08:45 GMT
-ETag: "35298d-2fea-d8f4cd40"
-Accept-Ranges: bytes
-Content-Length: 37
-Content-Range: bytes 1165-1201/12266
-Content-Type: text/html; charset=ISO-8859-1
-Connection: close
-
-"DRAKMA (Queen of Cosmic Greed)
-HUNCHENTOOT (The Giant Spider)"
-
- -
 

Download and installation

- -Drakma together with this documentation can be downloaded -from http://weitz.de/files/drakma.tar.gz. -The current version is 0.6.2. Drakma can be installed -via ASDF and depends on the -open source -libraries CL-BASE64 (use -3.3.2 or higher to avoid an unneeded dependency -on KMRCL), Puri, and Chunga. If -you're not using LispWorks, you'll also -need trivial-sockets -and (except for AllegroCL) CL+SSL. -Try to use the newest versions of all these libraries - use the CVS -versions if in doubt. Installation -via asdf-install -should also be possible, and -there's a port for Gentoo -Linux thanks to Matthew Kennedy. -

-For SSL, you -will need to -have the -corresponding C libraries as well. You'll usually have them -already unless you're on Windows. -

-Luís Oliveira maintains a darcs -repository of Drakma -at http://common-lisp.net/~loliveira/ediware/. - -
 

Support and mailing lists

- -For questions, bug reports, feature requests, improvements, or patches -please use the drakma-devel -mailing list. If you want to be notified about future releases -subscribe to the drakma-announce -mailing list. These mailing lists were made available thanks to -the services of common-lisp.net. - - -
 

The Drakma dictionary

- -

The request

- -The HTTP-REQUEST function is -the heart of Drakma. It is used to send requests to web servers and -will either return the message body of the server's reply or (if the -user so wishes) a stream one can read from. The wealth of keyword -parameters might look a bit intimidating first, but you will rarely -need more than two or three of them - the default behaviour of Drakma -is (hopefully) designed to do The Right Thing[TM] in most cases. -

-You can use -the *HEADER-STREAM* -variable to debug requests handled by Drakma in a way similar -to LiveHTTPHeaders. - - - -


[Function]
http-request uri &key protocol method force-ssl parameters form-data content content-length content-type cookie-jar basic-authorization user-agent accept proxy proxy-basic-authorization additional-headers redirect redirect-methods auto-referer keep-alive close external-format-out external-format-in force-binary want-stream stream connection-timeout read-timeout write-timeout
=> body-or-stream, status-code, headers, uri, stream, must-close
-


- -Sends an HTTP request to a web server and returns its reply. -uri is where the request is sent to, and it is -either a string denoting -a uniform -resource identifier or -a PURI:URI -object. The scheme of uri must be 'http' or -'https'. The function returns six values - the body of the -reply (but see below), the status code as an integer, -an alist -of the headers sent by the server where for each element -the car -(the name of the header) is a keyword and -the cdr -(the value of the header) is a string, the URI the reply comes from -(which might be different from the URI the request was sent to in case -of redirects), the stream the reply was read -from, and -a generalized -boolean which denotes whether the stream should be closed (and -which you can usually ignore). -

-protocol is the HTTP protocol which is going to be used in the -request line, it must be one of the keywords :HTTP/1.0 or -:HTTP/1.1 (the default). method is the method used in the -request line, -a keyword -(like :GET or :HEAD) denoting a -valid HTTP/1.1 -or WebDAV request method. -Additionally, you can also use the pseudo method :OPTIONS* which is like -:OPTIONS but means that an "OPTIONS *" -request line will be sent, i.e. the URI's path and query parts will be -ignored. -

-If force-ssl is true, -SSL -will be attached to the socket stream which connects Drakma with the -web server. Usually, you don't have to provide this argument, as SSL -will be attached anyway if the scheme of uri is 'https'. -

-parameters is -an alist -of name/value pairs -(the car -and -the cdr -each being a string) which denotes the parameters which are added to -the query part of the URI or (in the case of a POST request) comprise -the request body. (But -see content below.) The -name/value pairs -are URL-encoded -using the external format external-format-out -before they are sent to the server, unless form-data is true in which -case the POST request body is sent -as multipart/form-data -using -external-format-out. The values of the parameters alist can also be -pathnames -or lists where the first element is a pathname. These values denote -files which should be sent as part of the request body, i.e. if such -file designators are present in parameters, the -content type of the request -is always multipart/form-data. If the value -denoting a file is a list, the part of the list behind the pathname is -treated as -a plist -which can be used to optionally specify a content type (the default is -"application/octet-stream") and/or a filename (the default is the -result of -applying FILE-NAMESTRING -to the pathname) for the file. So, for example, a full file upload -request could look like this: -

-(http-request "http://www.whatever.com/file_upload/"
-              :method :post
-              ;; the following line is only needed if the receiving server doesn't accept
-              ;; chunked transfer encoding (like for example Apache 1.x)
-              :content-length t
-              :parameters '(("file1" #p"/tmp/top_secret_stuff.doc" :content-type "application/msword" :filename "upload.doc")
-                            ("file2" . #p"/tmp/portrait.jpg")
-                            ("lname" . "Duck") ("fname" . "Donald")))
-
-

-external-format-out (the default is the value of *DRAKMA-DEFAULT-EXTERNAL-FORMAT*) must be the name of a FLEXI-STREAMS external -format. -

content, if not NIL, is -used as the request body - parameters is ignored -in this case. content can be a string, a -sequence of octets, a pathname, an open binary input stream, or a -function -designator. If content is a sequence, it will -be directly sent to the server (using external-format-out in the case of strings). If content is a -pathname, the binary contents of the corresponding file will be sent -to the server. If content is a stream, everything -that can be read from the stream -until EOF -will be sent to the server. If content is a -function designator, the corresponding function will be called with -one argument, the stream to the server, to which it should send data. -

-Finally, content can also be the -keyword :CONTINUATION in which -case HTTP-REQUEST returns -only one value - a "continuation" function. This function has one -required argument and one optional argument. The first argument will -be interpreted like content above (but it cannot -be a keyword), i.e. it will be sent to the server according to its -type. If the second argument is true, the continuation function can -be called again to send more content, if it is NIL, the -continuation function returns -what HTTP-REQUEST would have -returned. See above for an -example on how to use a continuation function and different types of -content. -

-If content is a sequence, Drakma will -use LENGTH -to determine its length and will use the result for the -'Content-Length' header sent to the server. You can overwrite this -with the content-length parameter -(a non-negative integer) which you can also use for the cases where -Drakma can't or won't determine the content length itself. You can -also explicitely provide a content-length argument -of NIL which will imply that no 'Content-Length' header -will be sent even if Drakma could compute the value. If no -'Content-Length' header is sent, Drakma will -use chunked encoding to send the -content body. Note that this will not work with some older web -servers. -

-A non-NIL content-length argument -means that Drakma must build the request body in RAM and -compute the content length even if it would have otherwise used -chunked encoding - for example in the case of file uploads. A special -case is the value T -for content-length which means that Drakma should -compute the content length after building the request body. -

-content-type is the -corresponding 'Content-Type' header to be sent and will be ignored -unless content is provided as well. -

-Note that a -query already contained in uri will always be sent -with the request line anyway in addition to other parameters sent by -Drakma. -

-cookie-jar is a cookie -jar containing cookies which will potentially be sent to the -server (if the domain matches, if they haven't expired, etc.) - this -cookie jar will be modified according to the 'Set-Cookie' header(s) -sent back by the server. -

-basic-authorization, if not NIL, -should be a list of two strings (username and password) which will be -sent to the server for basic -authorization. user-agent, if -not NIL, denotes which 'User-Agent' header will be sent -with the request. It can be one of the keywords :DRAKMA -(the -default), :FIREFOX, :EXPLORER, :OPERA, -or -:SAFARI which denote the current version of Drakma or, in -the latter four cases, a fixed string corresponding to a more or less -recent (as of August 2006) version of the corresponding browser. Or -it can be a string which is used -directly. accept, if not NIL, is the -'Accept' header sent - the default is "*/*". -

-If proxy is not NIL, it should be a -string denoting -a proxy server -through which the request should be sent. Or it can be a list of two -values - a string denoting the proxy server and an integer denoting -the port to use (which will default to 80 otherwise). -proxy-basic-authorization is used like basic-authorization, but for -the proxy, and only if proxy is true. -

-additional-headers is a -name/value alist -(like parameters) of additional HTTP headers which -should be sent with the request. -

-If redirect is -not NIL, it must be a non-negative integer -or T. If redirect is true, Drakma -will follow redirects (return codes 301, 302, 303, or 307) -unless redirect is 0. -If redirect is an integer, it will be decreased -by 1 with each redirect. Drakma will only follow -redirects if method is a member of the list redirect-methods the -initial value of which is (:GET :HEAD). -Furthermore, if auto-referer is true when following redirects, -Drakma will populate the 'Referer' (sic!) header with the URI that -triggered the redirection, overwriting an existing 'Referer' -header (in additional-headers) if necessary. -

-If keep-alive is T, the server will -be asked to keep the connection alive, i.e. not to close it after the -reply has been sent. (Note that -this not necessary -if both the client and the server use HTTP 1.1.) -If close is T, the server is -explicitely asked to close the connection after the reply has been -sent. keep-alive and close -are obviously mutually -exclusive. The default for close is T, the default for keep-alive is NIL. -

-HTTP-REQUEST will always -close the stream to the server before it returns unless -want-stream is true or if the headers exchanged -between Drakma and the server determine that the connection will be -kept alive - for example if both client and server used the -HTTP 1.1 protocol and no -explicit "Connection: close" header was sent. In -these cases you will have to close the stream manually. -

-If the message body sent by the server has a -text content -type, Drakma will try to return it as a Lisp string. It'll first -check if the 'Content-Type' header denotes an encoding (charset) to be -used, or otherwise it will use -the external-format-in (the default is the value -of *DRAKMA-DEFAULT-EXTERNAL-FORMAT*) -argument. The body is decoded -using FLEXI-STREAMS. If -FLEXI-STREAMS doesn't know the external format, the body is returned -as an array of octets. -If the message body doesn't have a text content type or if -force-binary is true, the body is always returned -as an array of octets. -

-If want-stream is true, the -message body is not read and instead the (open) socket stream -is returned as the first return value. If the sixth return value -(must-close) -of HTTP-REQUEST is true, -Drakma deduced from the reply headers that the server will close the -stream on its side, so you can't re-use it - you'll have to close it -instead. Of course, no matter what the sixth return value is, it's -alway your responsibility to close the stream once you're done with -it. The -stream returned is a flexi -stream with a chunked stream -as its underlying stream. -

-Drakma will usually -create a -new socket connection for each HTTP request. However, you can use -the stream argument to provide an open socket stream which should be -re-used instead. stream must be a stream returned by a previous invocation of -HTTP-REQUEST where the sixth return value wasn't true. Obviously, it -must also be connected to the correct server and at the right position -(i.e. the message body, if any, must have been read). Drakma will -never attach SSL to a stream provided as the stream argument. -

-connection-timeout is the time (in seconds) Drakma -will wait until it considers an attempt to connect to a server as a -failure. read-timeout -and write-timeout are the read and write timeouts -(in seconds) for the socket stream to the server. All three timeout -arguments can also be NIL (meaning no timeout), and they -don't apply if an existing stream is re-used. All timeout keyword -arguments are only available for -LispWorks, write-timeout is only available for -LispWorks 5.0 or higher. - -

- - - - - - - -


[Special variable]
*drakma-default-external-format* -


- -The default value for the two external format keyword arguments of -HTTP-REQUEST. The value of -this variable will be interpreted -by FLEXI-STREAMS. The -initial value is the keyword :LATIN-1. (Note that Drakma -binds *DEFAULT-EOL-STYLE* -to :LF.) - -
- - - - - - -


[Special variable]
*header-stream* -


- -If this variable is not NIL, it should be bound to a -stream to which incoming and outgoing headers will be written for -debugging purposes. - -
- - - -

Cookies

- -HTTP-REQUEST can deal -with cookies if -it gets a cookie jar, a collection -of COOKIE objects, as -its cookie-jar argument. Cookies sent by the web -server will be added to the cookie jar (or updated) if appropriate and -cookies already in the cookie jar will be sent to the server together -with the request. -

-Drakma will never remove cookies from a cookie jar -automatically - you have to do it manually -using DELETE-OLD-COOKIES. - - - -


[Standard class]
cookie -


- -Elements of this class -represent HTTP -cookies. If you need to create your own cookies, you should -use MAKE-INSTANCE -with the -initargs :NAME, :DOMAIN, :VALUE, :PATH, :EXPIRES, -and :SECUREP all of which are optional except for the -first two. The meaning of these initargs and the -corresponding accessors should be pretty clear if one looks at -the original -cookie specification. - -
-DRAKMA-USER 18 > (make-instance 'cookie :name "Foo" 
-                                        :value "Bar"
-                                        :expires (+ (get-universal-time) 3600)
-                                        :domain ".weitz.de")
-#<COOKIE Foo=Bar; expires=Sat, 26-08-2006 23:14:27 GMT; path=/; domain=.weitz.de>
-
- -
- - - - -


[Specialized accessors]
cookie-name (cookie cookie) => name -
(setf (cookie-name (cookie cookie)) name) -
cookie-value (cookie cookie) => value -
(setf (cookie-value (cookie cookie)) value) -
cookie-domain (cookie cookie) => domain -
(setf (cookie-domain (cookie cookie)) domain) -
cookie-path (cookie cookie) => path -
(setf (cookie-path (cookie cookie)) path) -
cookie-expires (cookie cookie) => expiry -
(setf (cookie-expires (cookie cookie)) expiry) -
cookie-securep (cookie cookie) => securep -
(setf (cookie-securep (cookie cookie)) securep) -


- -These -are accessors -to get and set the corresponding slots of -a COOKIE object. Note that expiry is a universal time -and securep is a generalized -boolean. All other values are strings. - -
- - - -


[Standard class]
cookie-jar -


- -An object of this class encapsulates a collection (a list, actually) -of COOKIE objects. You create a -new cookie jar with -(MAKE-INSTANCE 'COOKIE-JAR) -where you can optionally provide a list -of COOKIE objects with -the :COOKIES initarg. The cookies in a cookie jar are -accessed -with COOKIE-JAR-COOKIES. - -
- - - - - - -


[Specialized accessor]
cookie-jar-cookies (cookie-jar cookie-jar) => list -
(setf (cookie-jar-cookies (cookie-jar cookie-jar)) list) -


- -This accessor -is used to get and set the cookies comprised in a cookie -jar. list is a list -of COOKIE objects. -

-Note that list should not contain two cookies which are equal according to COOKIE=. - -

- - - - - - -


[Function]
cookie= cookie1 cookie2 => result -


- -Returns true -if the cookies cookie1 -and cookie2 are equal. Two cookies are considered -to be equal if their names and paths are equal. - -
- - - - - - -


[Function]
delete-old-cookies cookie-jar => cookie-jar -


- -Removes all cookies from the cookie -jar cookie-jar which have either expired or -which don't have an expiry date. - -
- - - -

Headers

- -This section assembles a couple of convenience functions which can be -used to access information returned as the third -value (headers) -of HTTP-REQUEST. -

-Note that if the header -sends multiple headers -with the same name, these are comprised into one entry by -HTTP-REQUEST where the values -are separated by commas. - - - -


[Function]
header-value name headers => value -


- -If headers is an alist of headers as returned by HTTP-REQUEST -and name is a keyword naming a header, this function returns the -corresponding value of this header (or NIL if it's not in -headers). -
-DRAKMA-USER 19 > (setq *header-stream* nil)
-NIL
-DRAKMA-USER 20 > (header-value :server
-                               (nth-value 2 (http-request "http://www.jalat.com/blogs/lisp?id=5")))
-"Hunchentoot 0.1.3 (TBNL 0.9.7)"
-
- -
- - - - - -


[Function]
split-tokens string => string-list -


- -Splits the string string into a list of substrings separated -by commas and optional whitespace. Empty substrings are -ignored. -
-DRAKMA-USER 21 > (split-tokens "chunked, identity")
-("chunked" "identity")
-
- -
- - - - - - -


[Function]
read-tokens-and-parameters string &key value-required-p => list -


- -Reads a comma-separated list -of tokens from the -string string. Each token can be followed by an -optional, semicolon-separated list -of attribute/value -pairs where the attributes are tokens followed by -a #\= character and a token or -a quoted string. -Returned is a list where each element is either a string (for a simple -token) or -a cons -of a string (the token) and -an alist -(the attribute/value pairs). If value-required-p -is NIL (the default is T), the value part -(including the #\= character) of each attribute/value -pair is optional. -

-An example of an HTTP header which uses a syntax which can be parsed -with this function is the 'Transfer-Encoding' header. -

-DRAKMA-USER 21 > (read-tokens-and-parameters "iso-8859-5, unicode-1-1;q=0.8")
-("iso-8859-5" ("unicode-1-1" ("q" . "0.8")))
-
- -
- - - - - -


[Function]
parameter-present-p name parameters => generalized-boolean -


- -If parameters is -an alist -of parameters (i.e. of attribute/value pairs) as returned by, for -example, READ-TOKENS-AND-PARAMETERS and name is a string naming a -parameter, this function returns the full parameter (name and value) - -or NIL if it's not in parameters. -
-DRAKMA-USER 23 > (parameter-present-p "frob" '(("charset" . "latin-1") ("frob" . "quux")))
-("frob" . "quux")
-
-DRAKMA-USER 24 > (parameter-present-p "foo" '(("charset" . "latin-1") ("frob" . "quux")))
-NIL
-
-
- - - - - - -


[Function]
parameter-value name parameters => value -


- -If parameters is an alist of parameters (i.e. of attribute/value pairs) as returned by, for -example, READ-TOKENS-AND-PARAMETERS and name is a string naming a -parameter, this function returns the value of this parameter - or -NIL if it's not in parameters. -
-DRAKMA-USER 25 > (parameter-value "frob" '(("charset" . "latin-1") ("frob" . "quux")))
-"quux"
-
-DRAKMA-USER 26 > (parameter-value "foo" '(("charset" . "latin-1") ("frob" . "quux")))
-NIL
-
-
- - - - - -


[Function]
get-content-type headers => type, subtype, parameters -


- -Reads and parses a 'Content-Type' header and returns it as -three values - the type, the subtype, and an alist (possibly -empty) of name/value pairs for the optional parameters. headers -is supposed to be an alist of HTTP headers as returned by -HTTP-REQUEST. Returns NIL if there is no 'Content-Type' header amongst -headers. -
-DRAKMA-USER 27 > (get-content-type 
-                  (nth-value 2 (http-request "http://weitz.de/")))
-"text"
-"html"
-(("charset" . "iso-8859-1"))
-
-
- - - - - - -
 

Acknowledgements

- -Initial versions of Drakma used code -from ACL-COMPAT, -specifically the chunking code from Jochen Schmidt. (This has been replaced by Chunga.) -The API of -Drakma's HTTP-REQUEST was -inspired by John -Foderaro's DO-HTTP-REQUEST. -And greetings to Bob Hutchinson who -already anticipated this -library in 2005... :) - -

-This documentation was prepared with DOCUMENTATION-TEMPLATE. -

-

-$Header: /usr/local/cvsrep/drakma/doc/index.html,v 1.63 2007/03/09 08:39:28 edi Exp $ -

BACK TO MY HOMEPAGE - - - + + + + + + DRAKMA - A Common Lisp web client + + + + + +

DRAKMA - A Common Lisp web client

+ +
+
 

Abstract

+ +Drakma is a fully-featured web client (implemented in Common Lisp) +that knows how to handle HTTP/1.1 +chunking, persistent +connections, re-usable +sockets, SSL, continuable +uploads, file uploads, cookies, and other +things. And it's probably a result of +my NIH +syndrome... +

+Drakma was developed and tested +with LispWorks, but it should +also work with a couple of other Common Lisp implementations depending +on the supporting libraries. Some tests +with SBCL seem to confirm this. +

+The code comes with +a BSD-style +license so you can basically do with it whatever you want. + +

+Download shortcut: http://weitz.de/files/drakma.tar.gz. +

+ +
 

Contents

+
    +
  1. Examples +
  2. Download and installation +
  3. Support and mailing lists +
  4. The Drakma dictionary +
      +
    1. The request +
        +
      1. http-request +
      2. *drakma-default-external-format* +
      3. *header-stream* +
      +
    2. Cookies +
        +
      1. cookie +
      2. cookie-name +
      3. cookie-value +
      4. cookie-domain +
      5. cookie-path +
      6. cookie-expires +
      7. cookie-securep +
      8. cookie-jar +
      9. cookie-jar-cookies +
      10. cookie= +
      11. delete-old-cookies +
      +
    3. Headers +
        +
      1. header-value +
      2. split-tokens +
      3. read-tokens-and-parameters +
      4. parameter-present-p +
      5. parameter-value +
      6. get-content-type +
      +
    +
  5. Acknowledgements +
+ +
 

Examples

+ +Here's an example session with Drakma which demonstrates some of its +features. (Some linebreaks were added or removed to enhance +legibility.) Note that this doesn't necessarily reflect the current +version of Drakma. The examples should work nevertheless. + +
+;; create a log file of this sessions
+CL-USER 1 > (dribble "/tmp/drakma_dribble")
+; Loading C:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\ccl\dribble.ofasl on demand...
+
+;; load Drakma
+CL-USER 2 > (asdf:oos 'asdf:load-op :drakma)
+; loading system definition from c:\home\lisp\drakma\drakma.asd into
+; #<The ASDF0 package, 0/16 internal, 0/16 external>
+; Loading text file c:\home\lisp\drakma\drakma.asd
+; registering #<SYSTEM :DRAKMA 21D6D24F> as DRAKMA
+;; Creating system COMMON-LISP-USER::DRAKMA
+; loading system definition from c:\home\lisp\chunga\chunga.asd into
+; #<The ASDF0 package, 0/16 internal, 0/16 external>
+; Loading text file c:\home\lisp\chunga\chunga.asd
+; registering #<SYSTEM :CHUNGA 200B12A3> as CHUNGA
+;; Creating system COMMON-LISP-USER::CHUNGA
+; loading system definition from c:\home\lisp\flexi-streams\flexi-streams.asd into
+; #<The ASDF0 package, 0/16 internal, 0/16 external>
+; Loading text file c:\home\lisp\flexi-streams\flexi-streams.asd
+; registering #<SYSTEM :FLEXI-STREAMS 200E8017> as FLEXI-STREAMS
+;; Creating system COMMON-LISP-USER::FLEXI-STREAMS
+; loading system definition from c:\home\lisp\trivial-gray-streams\trivial-gray-streams.asd into
+; #<The ASDF0 package, 0/16 internal, 0/16 external>
+; Loading text file c:\home\lisp\trivial-gray-streams\trivial-gray-streams.asd
+; registering #<SYSTEM :TRIVIAL-GRAY-STREAMS 21D6741F> as TRIVIAL-GRAY-STREAMS
+;; Creating system COMMON-LISP-USER::TRIVIAL-GRAY-STREAMS
+; loading system definition from c:\home\lisp\cl-base64-3.3.2\cl-base64.asd into
+; #<The ASDF0 package, 0/16 internal, 0/16 external>
+; Loading text file c:\home\lisp\cl-base64-3.3.2\cl-base64.asd
+; registering #<SYSTEM CL-BASE64 21D6D277> as CL-BASE64
+;; Creating system COMMON-LISP-USER::CL-BASE64
+; registering #<SYSTEM CL-BASE64-TESTS 2009701B> as CL-BASE64-TESTS
+;; Creating system COMMON-LISP-USER::CL-BASE64-TESTS
+; loading system definition from c:\home\lisp\puri-1.5\puri.asd into
+; #<The ASDF0 package, 0/16 internal, 0/16 external>
+; Loading text file c:\home\lisp\puri-1.5\puri.asd
+; registering #<SYSTEM PURI 21D6B093> as PURI
+;; Creating system COMMON-LISP-USER::PURI
+; registering #<SYSTEM PURI-TESTS 200CFEEF> as PURI-TESTS
+;; Creating system COMMON-LISP-USER::PURI-TESTS
+; Loading fasl file c:\home\lisp\trivial-gray-streams\package.ofasl
+; Loading fasl file c:\home\lisp\trivial-gray-streams\mixin.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\packages.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\ascii.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\iso-8859.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\code-pages.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\specials.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\util.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\external-format.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\in-memory.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\stream.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\output.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\input.ofasl
+; Loading fasl file c:\home\lisp\flexi-streams\strings.ofasl
+; Loading fasl file c:\home\lisp\chunga\packages.ofasl
+; Loading fasl file c:\home\lisp\chunga\specials.ofasl
+; Loading fasl file c:\home\lisp\chunga\util.ofasl
+; Loading fasl file c:\home\lisp\chunga\read.ofasl
+; Loading fasl file c:\home\lisp\chunga\streams.ofasl
+; Loading fasl file c:\home\lisp\chunga\input.ofasl
+; Loading fasl file c:\home\lisp\chunga\output.ofasl
+; Loading fasl file c:\home\lisp\cl-base64-3.3.2\package.ofasl
+; Loading fasl file c:\home\lisp\cl-base64-3.3.2\encode.ofasl
+; Loading fasl file c:\home\lisp\cl-base64-3.3.2\decode.ofasl
+; Loading fasl file c:\home\lisp\puri-1.5\src.ofasl
+; Loading fasl file c:\home\lisp\drakma\packages.ofasl
+; Loading fasl file c:\home\lisp\drakma\specials.ofasl
+; Loading fasl file c:\home\lisp\drakma\util.ofasl
+; Loading c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\comm-defsys.lisp on demand...
+;; Creating system COMM
+
+;  Loading text file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\comm-pkg.lisp
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\sockets.ofasl
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl-constants.ofasl
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl-foreign-types.ofasl
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl.ofasl
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\load-on-demand\processes\ssl-certs.ofasl
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0001.ofasl
+; Loaded public patch COMM 1.1
+
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0002.ofasl
+; Loaded public patch COMM 1.2
+
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0003.ofasl
+; Loaded public patch COMM 1.3
+
+;  Loading fasl file c:\Program Files\LispWorks\lib\5-0-0-0\patches\comm\0001\0004.ofasl
+; Loaded public patch COMM 1.4
+
+; Loading fasl file c:\home\lisp\drakma\read.ofasl
+; Loading fasl file c:\home\lisp\drakma\cookies.ofasl
+; Loading fasl file c:\home\lisp\drakma\request.ofasl
+NIL
+
+;; create a package to work in
+CL-USER 3 > (defpackage :drakma-user (:use :cl :drakma))
+#<The DRAKMA-USER package, 0/16 internal, 0/16 external>
+
+;; switch to this package
+CL-USER 4 > (in-package :drakma-user)
+#<The DRAKMA-USER package, 0/16 internal, 0/16 external>
+
+;; log headers, so we can see what happens -
+;; output to *HEADER-STREAM* will be shown in green below
+DRAKMA-USER 5 > (setq *header-stream* *standard-output*)
+#<Broadcast stream to (#<Echo Stream Input = #<EDITOR::RUBBER-STREAM #<EDITOR:BUFFER CAPI interactive-pane 2> 2198ECD7>,
+                                     Output = #<STREAM::LATIN-1-FILE-STREAM c:\tmp\drakma_dribble>>
+                       #<EDITOR::RUBBER-STREAM #<EDITOR:BUFFER CAPI interactive-pane 2> 2198ECD7>)>
+
+;; note how Drakma automatically follows the 301 redirect and how the fourth return value shows the new URI
+DRAKMA-USER 6 > (http-request "http://lisp.org/")
+GET / HTTP/1.1
+Host: lisp.org
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 301  Moved Permanently
+Date: Sat, 26 Aug 2006 15:46:31 GMT
+Connection: Close
+Server: AllegroServe/1.2.37
+Transfer-Encoding: chunked
+LOCATION: /index.html
+
+GET /index.html HTTP/1.1
+Host: lisp.org
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 200  OK
+Date: Sat, 26 Aug 2006 15:46:32 GMT
+Connection: Close
+Server: AllegroServe/1.2.37
+Content-Type: text/html
+Content-Length: 82
+LAST-MODIFIED: Mon, 16 Feb 2004 09:30:02 GMT
+
+"<title>redirect...</title>
+<meta http-equiv=\"Refresh\" content=\"0; url=/alu/home\">
+"
+200
+((:DATE . "Sat, 26 Aug 2006 15:46:32 GMT")
+ (:CONNECTION . "Close")
+ (:SERVER . "AllegroServe/1.2.37")
+ (:CONTENT-TYPE . "text/html")
+ (:CONTENT-LENGTH . "82")
+ (:LAST-MODIFIED . "Mon, 16 Feb 2004 09:30:02 GMT"))
+#<URI http://lisp.org/index.html>
+#<FLEXI-STREAMS:FLEXI-IO-STREAM 201017D3>
+T
+
+;; here, Drakma automatically interprets the 'charset=utf-8' part correctly -
+;; might look a bit different in your listener depending on the font you've chosen
+DRAKMA-USER 7 > (subseq (http-request "http://www.cl.cam.ac.uk/~mgk25/ucs/examples/digraphs.txt") 0 298)
+GET /~mgk25/ucs/examples/digraphs.txt HTTP/1.1
+Host: www.cl.cam.ac.uk
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 16:02:56 GMT
+Server: Apache/1.3.37 (Unix) mod_ucam_webauth/1.2.2
+Last-Modified: Thu, 05 Jan 2006 20:49:55 GMT
+ETag: "17cd62-298-43bd8673"
+Accept-Ranges: bytes
+Content-Length: 664
+Connection: close
+Content-Type: text/plain; charset=utf-8
+
+"Latin Digraphs and Ligatures in ISO10646-1
+
+A short table of ligatures and digraphs follows. Some of these may not be
+ligatures/digraphs in the technical sense, (for example, æ is a seperate
+letter in English), but visually they behave that way.
+
+AÆE : U+00C6
+aæe : U+00E6
+ſßs : U+00DF
+IIJJ : U+0132"
+
+;; a vector of octets is returned for (non-text) binary data - a picture in this case
+DRAKMA-USER 8 > (http-request "http://zappa.com/favicon.ico")
+GET /favicon.ico HTTP/1.1
+Host: zappa.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 16:02:59 GMT
+Server: Apache/2.0.46 (Red Hat)
+Last-Modified: Fri, 17 Mar 2006 08:11:07 GMT
+ETag: "3a4080-b6-59d5bcc0"
+Accept-Ranges: bytes
+Content-Length: 182
+Connection: close
+Content-Type: image/gif
+
+#(71 73 70 56 57 97 17 0 17 0 179 1 0 150 151 153 255 255 255 37 37 36 112 114 115
+  201 202 204 0 0 0 80 83 84 26 28 26 230 231 231 249 249 249 12 13 14 219 221 222
+  18 21 22 239 240 241 52 52 54 64 66 66 33 249 4 1 0 0 1 0 44 0 0 0 0 17 0 17 0 0
+  4 99 48 200 73 107 109 54 172 101 129 120 196 180 12 12 51 80 64 161 42 3 48 28
+  170 106 72 141 16 223 120 113 166 121 95 0 14 95 239 33 236 41 98 10 129 114 185
+  188 29 127 25 201 224 73 60 4 8 0 130 22 59 64 52 96 135 148 35 96 80 152 159 186
+  192 64 183 112 0 200 61 65 0 1 192 76 214 185 113 102 241 88 26 90 8 81 18 8 94 
+  130 134 22 17 0 59)
+200
+((:DATE . "Sat, 26 Aug 2006 16:02:59 GMT")
+ (:SERVER . "Apache/2.0.46 (Red Hat)")
+ (:LAST-MODIFIED . "Fri, 17 Mar 2006 08:11:07 GMT")
+ (:ETAG . "\"3a4080-b6-59d5bcc0\"")
+ (:ACCEPT-RANGES . "bytes")
+ (:CONTENT-LENGTH . "182")
+ (:CONNECTION . "close")
+ (:CONTENT-TYPE . "image/gif"))
+#<URI http://zappa.com/favicon.ico>
+#<FLEXI-STREAMS:FLEXI-IO-STREAM 200D59BF>
+T
+
+;; a secure connection (see below) -
+;; also note that the server uses chunked transfer encoding for its reply
+DRAKMA-USER 9 > (ppcre:scan-to-strings "(?s)You have.*your data."
+                                       (http-request "https://www.fortify.net/cgi/ssl_2.pl"))
+GET /cgi/ssl_2.pl HTTP/1.1
+Host: www.fortify.net
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 16:10:06 GMT
+Server: Apache
+Connection: close
+Transfer-Encoding: chunked
+Content-Type: text/html
+
+
+"You have connected to this web server using the DHE-RSA-AES256-SHA encryption cipher
+ with a key length of 256 bits.
+ <p>
+ This is a high-grade encryption connection, regarded by most experts as being suitable
+ for sending or receiving even the most sensitive or valuable information
+ across a network.
+ <p>
+ In a crude analogy, using this cipher is similar to sending or storing your data inside
+ a high quality safe - compared to an export-grade cipher which is similar to using
+ a paper envelope to protect your data."
+#()
+
+;; using a different 'User-Agent' header
+DRAKMA-USER 10 > (ppcre:regex-replace-all
+                  "<.*?>"
+                  (ppcre:scan-to-strings "(?s)Your browser reports.*?</table>" 
+                                         (http-request "http://bcheck.scanit.be/bcheck/"
+                                                       :user-agent :explorer))
+                  "")
+GET /bcheck/ HTTP/1.1
+Host: bcheck.scanit.be
+User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
+Accept: */*
+Connection: close
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 16:21:50 GMT
+Server: Apache
+Connection: close
+Transfer-Encoding: chunked
+Content-Type: text/html
+
+
+"Your browser reports to be:
+
+Browser name: MSIE
+Version: 6.0
+Platform: Windows NT 5.1
+"
+
+;; sending parameters in a POST request and working with cookies -
+;; note how Drakma sends the cookie back in the second request
+DRAKMA-USER 11 > (let ((cookie-jar (make-instance 'cookie-jar)))
+                   (http-request "http://www.phpsecurepages.com/test/test.php"
+                                 :method :post
+                                 :parameters '(("entered_login" . "test")
+                                               ("entered_password" . "test"))
+                                 :cookie-jar cookie-jar)
+                   (http-request "http://www.phpsecurepages.com/test/test2.php"
+                                 :cookie-jar cookie-jar)
+                   (cookie-jar-cookies cookie-jar))
+POST /test/test.php HTTP/1.1
+Host: www.phpsecurepages.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Content-Length: 40
+Content-Type: application/x-www-form-urlencoded
+Connection: close
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 18:26:17 GMT
+Server: Apache/2.0.51 (Fedora)
+X-Powered-By: PHP/4.3.10
+Expires: Thu, 19 Nov 1981 08:52:00 GMT
+Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
+Pragma: no-cache
+Set-Cookie: PHPSESSID=3ce33aa3e326ab4bf5da7feecc3248b4; path=/
+Connection: close
+Transfer-Encoding: chunked
+Content-Type: text/html
+
+
+GET /test/test2.php HTTP/1.1
+Host: www.phpsecurepages.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Cookie: PHPSESSID=3ce33aa3e326ab4bf5da7feecc3248b4
+Connection: close
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 18:26:18 GMT
+Server: Apache/2.0.51 (Fedora)
+X-Powered-By: PHP/4.3.10
+Expires: Thu, 19 Nov 1981 08:52:00 GMT
+Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
+Pragma: no-cache
+Connection: close
+Transfer-Encoding: chunked
+Content-Type: text/html
+
+
+(#<COOKIE PHPSESSID=3ce33aa3e326ab4bf5da7feecc3248b4; path=/; domain=www.phpsecurepages.com>)
+
+;; now we are going to re-use a socket for the second connection to the same server
+;; this will also work with chunked encoding
+DRAKMA-USER 12 > (let ((stream (nth-value 4 (http-request "http://www.lispworks.com/" :close nil))))
+                   (nth-value 2 (http-request "http://www.lispworks.com/success-stories/index.html"
+                                              :stream stream)))
+GET / HTTP/1.1
+Host: www.lispworks.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 18:34:20 GMT
+Server: Apache/1.3.37 Ben-SSL/1.57 (Unix)
+Last-Modified: Tue, 08 Aug 2006 18:20:49 GMT
+ETag: "28ee4f0-22db-44d8d601"
+Accept-Ranges: bytes
+Content-Length: 8923
+Content-Type: text/html
+
+GET /success-stories/index.html HTTP/1.1
+Host: www.lispworks.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 200 OK
+Date: Sat, 26 Aug 2006 18:34:20 GMT
+Server: Apache/1.3.37 Ben-SSL/1.57 (Unix)
+Last-Modified: Tue, 08 Aug 2006 18:22:19 GMT
+ETag: "28f3f42-2325-44d8d65b"
+Accept-Ranges: bytes
+Content-Length: 8997
+Connection: close
+Content-Type: text/html
+
+((:DATE . "Sat, 26 Aug 2006 18:34:20 GMT")
+ (:SERVER . "Apache/1.3.37 Ben-SSL/1.57 (Unix)")
+ (:LAST-MODIFIED . "Tue, 08 Aug 2006 18:22:19 GMT")
+ (:ETAG . "\"28f3f42-2325-44d8d65b\"")
+ (:ACCEPT-RANGES . "bytes")
+ (:CONTENT-LENGTH . "8997")
+ (:CONNECTION . "close")
+ (:CONTENT-TYPE . "text/html"))
+
+;; testing basic authorization against a local Hunchentoot server
+DRAKMA-USER 13 > (nth-value 1 (http-request "http://localhost:9000/tbnl/test/authorization.html"))
+GET /tbnl/test/authorization.html HTTP/1.1
+Host: localhost:9000
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 401 Authorization Required
+Content-Length: 563
+Content-Type: text/html; charset=iso-8859-1
+Date: Sat, 26 Aug 2006 18:38:58 GMT
+Server: Hunchentoot 0.1.5 (TBNL 0.10.0)
+Connection: Close
+WWW-Authenticate: Basic realm="TBNL"
+
+401
+
+DRAKMA-USER 14 > (nth-value 1 (http-request "http://localhost:9000/tbnl/test/authorization.html"
+                                            :basic-authorization '("nanook" "igloo")))
+GET /tbnl/test/authorization.html HTTP/1.1
+Host: localhost:9000
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Authorization: Basic bmFub29rOmlnbG9v
+Accept: */*
+Connection: close
+
+HTTP/1.1 200 OK
+Content-Length: 884
+Content-Type: text/html; charset=iso-8859-1
+Date: Sat, 26 Aug 2006 18:39:19 GMT
+Server: Hunchentoot 0.1.5 (TBNL 0.10.0)
+Connection: Close
+
+200
+
+;; now we ask Drakma to return a stream and read from it directly
+DRAKMA-USER 15 > (let ((stream (http-request "http://www.jalat.com/blogs/lisp?id=3" 
+                                             :want-stream t)))
+                   (loop for i below 41
+                         for line = (read-line stream)
+                         when (> i 35)
+                         do (write-line line))
+                   (close stream)
+                   (values))
+GET /blogs/lisp?id=3 HTTP/1.1
+Host: www.jalat.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+
+HTTP/1.1 200 OK
+Content-Length: 21453
+Content-Type: text/html; charset=iso-8859-1
+Date: Sat, 26 Aug 2006 19:53:37 GMT
+Server: Hunchentoot 0.1.3 (TBNL 0.9.7)
+Connection: Close
+
+Bill Clementson has <a
+href="http://bc.tech.coop/blog/041111.html">written</a> about getting
+TBNL up and running with apache and mod_lisp. In this example I'm
+going to use <a href="http://weitz.de/hunchentoot/">hunchentoot</a>, a
+pure lisp web server by (again) Edi Weitz.
+
+;; let's test a POST request without content length and with chunked transfer encoding -
+;; we build the content in several steps using different types of data
+;; (note: doesn't work anymore, probably due to server changes)
+DRAKMA-USER 16 > (let ((temp-file (ensure-directories-exist #p"/tmp/quux.txt"))
+                       (continuation (http-request "http://meme.b9.com/login.html"
+                                                   :method :post
+                                                   :content :continuation)))
+                   (funcall continuation "username=" t)
+                   (funcall continuation (list (char-code #\n) (char-code #\a)) t)
+                   (funcall continuation (lambda (stream)
+                                           (write-char #\n stream)) t)
+                   (with-open-file (out temp-file
+                                        :direction :output
+                                        :if-does-not-exist :create
+                                        :if-exists :supersede)
+                     (write-string "ook" out))
+                   (funcall continuation temp-file t)
+                   (ppcre:scan-to-strings "(?i)[a-z ]+nanook[a-z .]+"
+                                          (funcall continuation "&password=igloo")))
+POST /login.html HTTP/1.1
+Host: meme.b9.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+Content-Type: application/x-www-form-urlencoded
+Transfer-Encoding: chunked
+
+HTTP/1.0 200 OK
+Date: Sat, 02 Sep 2006 00:25:24 GMT
+Connection: close
+Server: AllegroServe/1.2.45
+Content-Type: text/html
+Content-Length: 2922
+PRAGMA: no-cache
+CACHE-CONTROL: no-cache
+SET-COOKIE: meme=1834b91d26f9be983a0ed9ca; path=/
+
+"The username nanook is not in our database."
+#()
+
+;; finally, we send additional headers to ask for a range
+DRAKMA-USER 17 > (ppcre:regex-replace-all
+                  "<.*?>"
+                  (format nil "~A~A"
+                          (http-request "http://users.cableaz.com/~lantz/pages/hunchentoot.html"
+                                        :additional-headers '(("Range" . "bytes=959-999")))
+                          (http-request "http://users.cableaz.com/~lantz/pages/hunchentoot.html"
+                                        :additional-headers '(("Range" . "bytes=1165-1201"))))
+                  "")
+GET /~lantz/pages/hunchentoot.html HTTP/1.1
+Host: users.cableaz.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+Range: bytes=959-999
+
+HTTP/1.1 206 Partial Content
+Date: Sat, 26 Aug 2006 19:07:44 GMT
+Server: Apache/2.0.16 (Unix)
+Last-Modified: Sun, 24 Apr 2005 04:08:45 GMT
+ETag: "35298d-2fea-d8f4cd40"
+Accept-Ranges: bytes
+Content-Length: 41
+Content-Range: bytes 959-999/12266
+Content-Type: text/html; charset=ISO-8859-1
+Connection: close
+
+GET /~lantz/pages/hunchentoot.html HTTP/1.1
+Host: users.cableaz.com
+User-Agent: Drakma/0.3.0 (LispWorks 5.0.0; Windows NT; Windows XP: 5.1 (build 2600) Service Pack 2; http://weitz.de/drakma/)
+Accept: */*
+Connection: close
+Range: bytes=1165-1201
+
+HTTP/1.1 206 Partial Content
+Date: Sat, 26 Aug 2006 19:07:45 GMT
+Server: Apache/2.0.16 (Unix)
+Last-Modified: Sun, 24 Apr 2005 04:08:45 GMT
+ETag: "35298d-2fea-d8f4cd40"
+Accept-Ranges: bytes
+Content-Length: 37
+Content-Range: bytes 1165-1201/12266
+Content-Type: text/html; charset=ISO-8859-1
+Connection: close
+
+"DRAKMA (Queen of Cosmic Greed)
+HUNCHENTOOT (The Giant Spider)"
+
+ +
 

Download and installation

+ +Drakma together with this documentation can be downloaded +from http://weitz.de/files/drakma.tar.gz. +The current version is 0.7.0. Drakma can be installed +via ASDF and depends on the +open source +libraries CL-BASE64 (use +3.3.2 or higher to avoid an unneeded dependency +on KMRCL), Puri, and Chunga. If +you're not using LispWorks, you'll also +need usocket (0.3.2 or newer) +and (except for AllegroCL) CL+SSL. +Try to use the newest versions of all these libraries - use the CVS +versions if in doubt. Installation +via asdf-install +should also be possible, and +there's a port for Gentoo +Linux thanks to Matthew Kennedy. +

+For SSL, you +will need to +have the +corresponding C libraries as well. You'll usually have them +already unless you're on Windows. +

+Luís Oliveira maintains a darcs +repository of Drakma +at http://common-lisp.net/~loliveira/ediware/. + +
 

Support and mailing lists

+ +For questions, bug reports, feature requests, improvements, or patches +please use the drakma-devel +mailing list. If you want to be notified about future releases +subscribe to the drakma-announce +mailing list. These mailing lists were made available thanks to +the services of common-lisp.net. + + +
 

The Drakma dictionary

+ +

The request

+ +The HTTP-REQUEST function is +the heart of Drakma. It is used to send requests to web servers and +will either return the message body of the server's reply or (if the +user so wishes) a stream one can read from. The wealth of keyword +parameters might look a bit intimidating first, but you will rarely +need more than two or three of them - the default behaviour of Drakma +is (hopefully) designed to do The Right Thing[TM] in most cases. +

+You can use +the *HEADER-STREAM* +variable to debug requests handled by Drakma in a way similar +to LiveHTTPHeaders. + + + +


[Function]
http-request uri &key protocol method force-ssl parameters form-data content content-length content-type cookie-jar basic-authorization user-agent accept proxy proxy-basic-authorization additional-headers redirect redirect-methods auto-referer keep-alive close external-format-out external-format-in force-binary want-stream stream connection-timeout read-timeout write-timeout
=> body-or-stream, status-code, headers, uri, stream, must-close
+


+ +Sends an HTTP request to a web server and returns its reply. +uri is where the request is sent to, and it is +either a string denoting +a uniform +resource identifier or +a PURI:URI +object. The scheme of uri must be 'http' or +'https'. The function returns six values - the body of the +reply (but see below), the status code as an integer, +an alist +of the headers sent by the server where for each element +the car +(the name of the header) is a keyword and +the cdr +(the value of the header) is a string, the URI the reply comes from +(which might be different from the URI the request was sent to in case +of redirects), the stream the reply was read +from, and +a generalized +boolean which denotes whether the stream should be closed (and +which you can usually ignore). +

+protocol is the HTTP protocol which is going to be used in the +request line, it must be one of the keywords :HTTP/1.0 or +:HTTP/1.1 (the default). method is the method used in the +request line, +a keyword +(like :GET or :HEAD) denoting a +valid HTTP/1.1 +or WebDAV request method. +Additionally, you can also use the pseudo method :OPTIONS* which is like +:OPTIONS but means that an "OPTIONS *" +request line will be sent, i.e. the URI's path and query parts will be +ignored. +

+If force-ssl is true, +SSL +will be attached to the socket stream which connects Drakma with the +web server. Usually, you don't have to provide this argument, as SSL +will be attached anyway if the scheme of uri is 'https'. +

+parameters is +an alist +of name/value pairs +(the car +and +the cdr +each being a string) which denotes the parameters which are added to +the query part of the URI or (in the case of a POST request) comprise +the request body. (But +see content below.) The +name/value pairs +are URL-encoded +using the external format external-format-out +before they are sent to the server, unless form-data is true in which +case the POST request body is sent +as multipart/form-data +using +external-format-out. The values of the parameters alist can also be +pathnames +or lists where the first element is a pathname. These values denote +files which should be sent as part of the request body, i.e. if such +file designators are present in parameters, the +content type of the request +is always multipart/form-data. If the value +denoting a file is a list, the part of the list behind the pathname is +treated as +a plist +which can be used to optionally specify a content type (the default is +"application/octet-stream") and/or a filename (the default is the +result of +applying FILE-NAMESTRING +to the pathname) for the file. So, for example, a full file upload +request could look like this: +

+(http-request "http://www.whatever.com/file_upload/"
+              :method :post
+              ;; the following line is only needed if the receiving server doesn't accept
+              ;; chunked transfer encoding (like for example Apache 1.x)
+              :content-length t
+              :parameters '(("file1" #p"/tmp/top_secret_stuff.doc" :content-type "application/msword" :filename "upload.doc")
+                            ("file2" . #p"/tmp/portrait.jpg")
+                            ("lname" . "Duck") ("fname" . "Donald")))
+
+

+external-format-out (the default is the value of *DRAKMA-DEFAULT-EXTERNAL-FORMAT*) must be the name of a FLEXI-STREAMS external +format. +

content, if not NIL, is +used as the request body - parameters is ignored +in this case. content can be a string, a +sequence of octets, a pathname, an open binary input stream, or a +function +designator. If content is a sequence, it will +be directly sent to the server (using external-format-out in the case of strings). If content is a +pathname, the binary contents of the corresponding file will be sent +to the server. If content is a stream, everything +that can be read from the stream +until EOF +will be sent to the server. If content is a +function designator, the corresponding function will be called with +one argument, the stream to the server, to which it should send data. +

+Finally, content can also be the +keyword :CONTINUATION in which +case HTTP-REQUEST returns +only one value - a "continuation" function. This function has one +required argument and one optional argument. The first argument will +be interpreted like content above (but it cannot +be a keyword), i.e. it will be sent to the server according to its +type. If the second argument is true, the continuation function can +be called again to send more content, if it is NIL, the +continuation function returns +what HTTP-REQUEST would have +returned. See above for an +example on how to use a continuation function and different types of +content. +

+If content is a sequence, Drakma will +use LENGTH +to determine its length and will use the result for the +'Content-Length' header sent to the server. You can overwrite this +with the content-length parameter +(a non-negative integer) which you can also use for the cases where +Drakma can't or won't determine the content length itself. You can +also explicitely provide a content-length argument +of NIL which will imply that no 'Content-Length' header +will be sent even if Drakma could compute the value. If no +'Content-Length' header is sent, Drakma will +use chunked encoding to send the +content body. Note that this will not work with some older web +servers. +

+A non-NIL content-length argument +means that Drakma must build the request body in RAM and +compute the content length even if it would have otherwise used +chunked encoding - for example in the case of file uploads. A special +case is the value T +for content-length which means that Drakma should +compute the content length after building the request body. +

+content-type is the +corresponding 'Content-Type' header to be sent and will be ignored +unless content is provided as well. +

+Note that a +query already contained in uri will always be sent +with the request line anyway in addition to other parameters sent by +Drakma. +

+cookie-jar is a cookie +jar containing cookies which will potentially be sent to the +server (if the domain matches, if they haven't expired, etc.) - this +cookie jar will be modified according to the 'Set-Cookie' header(s) +sent back by the server. +

+basic-authorization, if not NIL, +should be a list of two strings (username and password) which will be +sent to the server for basic +authorization. user-agent, if +not NIL, denotes which 'User-Agent' header will be sent +with the request. It can be one of the keywords :DRAKMA +(the +default), :FIREFOX, :EXPLORER, :OPERA, +or +:SAFARI which denote the current version of Drakma or, in +the latter four cases, a fixed string corresponding to a more or less +recent (as of August 2006) version of the corresponding browser. Or +it can be a string which is used +directly. accept, if not NIL, is the +'Accept' header sent - the default is "*/*". +

+If proxy is not NIL, it should be a +string denoting +a proxy server +through which the request should be sent. Or it can be a list of two +values - a string denoting the proxy server and an integer denoting +the port to use (which will default to 80 otherwise). +proxy-basic-authorization is used like basic-authorization, but for +the proxy, and only if proxy is true. +

+additional-headers is a +name/value alist +(like parameters) of additional HTTP headers which +should be sent with the request. +

+If redirect is +not NIL, it must be a non-negative integer +or T. If redirect is true, Drakma +will follow redirects (return codes 301, 302, 303, or 307) +unless redirect is 0. +If redirect is an integer, it will be decreased +by 1 with each redirect. Drakma will only follow +redirects if method is a member of the list redirect-methods the +initial value of which is (:GET :HEAD). +Furthermore, if auto-referer is true when following redirects, +Drakma will populate the 'Referer' (sic!) header with the URI that +triggered the redirection, overwriting an existing 'Referer' +header (in additional-headers) if necessary. +

+If keep-alive is T, the server will +be asked to keep the connection alive, i.e. not to close it after the +reply has been sent. (Note that +this not necessary +if both the client and the server use HTTP 1.1.) +If close is T, the server is +explicitely asked to close the connection after the reply has been +sent. keep-alive and close +are obviously mutually +exclusive. The default for close is T, the default for keep-alive is NIL. +

+HTTP-REQUEST will always +close the stream to the server before it returns unless +want-stream is true or if the headers exchanged +between Drakma and the server determine that the connection will be +kept alive - for example if both client and server used the +HTTP 1.1 protocol and no +explicit "Connection: close" header was sent. In +these cases you will have to close the stream manually. +

+If the message body sent by the server has a +text content +type, Drakma will try to return it as a Lisp string. It'll first +check if the 'Content-Type' header denotes an encoding (charset) to be +used, or otherwise it will use +the external-format-in (the default is the value +of *DRAKMA-DEFAULT-EXTERNAL-FORMAT*) +argument. The body is decoded +using FLEXI-STREAMS. If +FLEXI-STREAMS doesn't know the external format, the body is returned +as an array of octets. +If the message body doesn't have a text content type or if +force-binary is true, the body is always returned +as an array of octets. +

+If want-stream is true, the +message body is not read and instead the (open) socket stream +is returned as the first return value. If the sixth return value +(must-close) +of HTTP-REQUEST is true, +Drakma deduced from the reply headers that the server will close the +stream on its side, so you can't re-use it - you'll have to close it +instead. Of course, no matter what the sixth return value is, it's +alway your responsibility to close the stream once you're done with +it. The +stream returned is a flexi +stream with a chunked stream +as its underlying stream. +

+Drakma will usually +create a +new socket connection for each HTTP request. However, you can use +the stream argument to provide an open socket stream which should be +re-used instead. stream must be a stream returned by a previous invocation of +HTTP-REQUEST where the sixth return value wasn't true. Obviously, it +must also be connected to the correct server and at the right position +(i.e. the message body, if any, must have been read). Drakma will +never attach SSL to a stream provided as the stream argument. +

+connection-timeout is the time (in seconds) Drakma +will wait until it considers an attempt to connect to a server as a +failure. read-timeout +and write-timeout are the read and write timeouts +(in seconds) for the socket stream to the server. All three timeout +arguments can also be NIL (meaning no timeout), and they +don't apply if an existing stream is re-used. All timeout keyword +arguments are only available for +LispWorks, write-timeout is only available for +LispWorks 5.0 or higher. + +

+ + + + + + + +


[Special variable]
*drakma-default-external-format* +


+ +The default value for the two external format keyword arguments of +HTTP-REQUEST. The value of +this variable will be interpreted +by FLEXI-STREAMS. The +initial value is the keyword :LATIN-1. (Note that Drakma +binds *DEFAULT-EOL-STYLE* +to :LF.) + +
+ + + + + + +


[Special variable]
*header-stream* +


+ +If this variable is not NIL, it should be bound to a +stream to which incoming and outgoing headers will be written for +debugging purposes. + +
+ + + +

Cookies

+ +HTTP-REQUEST can deal +with cookies if +it gets a cookie jar, a collection +of COOKIE objects, as +its cookie-jar argument. Cookies sent by the web +server will be added to the cookie jar (or updated) if appropriate and +cookies already in the cookie jar will be sent to the server together +with the request. +

+Drakma will never remove cookies from a cookie jar +automatically - you have to do it manually +using DELETE-OLD-COOKIES. + + + +


[Standard class]
cookie +


+ +Elements of this class +represent HTTP +cookies. If you need to create your own cookies, you should +use MAKE-INSTANCE +with the +initargs :NAME, :DOMAIN, :VALUE, :PATH, :EXPIRES, +and :SECUREP all of which are optional except for the +first two. The meaning of these initargs and the +corresponding accessors should be pretty clear if one looks at +the original +cookie specification. + +
+DRAKMA-USER 18 > (make-instance 'cookie :name "Foo" 
+                                        :value "Bar"
+                                        :expires (+ (get-universal-time) 3600)
+                                        :domain ".weitz.de")
+#<COOKIE Foo=Bar; expires=Sat, 26-08-2006 23:14:27 GMT; path=/; domain=.weitz.de>
+
+ +
+ + + + +


[Specialized accessors]
cookie-name (cookie cookie) => name +
(setf (cookie-name (cookie cookie)) name) +
cookie-value (cookie cookie) => value +
(setf (cookie-value (cookie cookie)) value) +
cookie-domain (cookie cookie) => domain +
(setf (cookie-domain (cookie cookie)) domain) +
cookie-path (cookie cookie) => path +
(setf (cookie-path (cookie cookie)) path) +
cookie-expires (cookie cookie) => expiry +
(setf (cookie-expires (cookie cookie)) expiry) +
cookie-securep (cookie cookie) => securep +
(setf (cookie-securep (cookie cookie)) securep) +


+ +These +are accessors +to get and set the corresponding slots of +a COOKIE object. Note that expiry is a universal time +and securep is a generalized +boolean. All other values are strings. + +
+ + + +


[Standard class]
cookie-jar +


+ +An object of this class encapsulates a collection (a list, actually) +of COOKIE objects. You create a +new cookie jar with +(MAKE-INSTANCE 'COOKIE-JAR) +where you can optionally provide a list +of COOKIE objects with +the :COOKIES initarg. The cookies in a cookie jar are +accessed +with COOKIE-JAR-COOKIES. + +
+ + + + + + +


[Specialized accessor]
cookie-jar-cookies (cookie-jar cookie-jar) => list +
(setf (cookie-jar-cookies (cookie-jar cookie-jar)) list) +


+ +This accessor +is used to get and set the cookies comprised in a cookie +jar. list is a list +of COOKIE objects. +

+Note that list should not contain two cookies which are equal according to COOKIE=. + +

+ + + + + + +


[Function]
cookie= cookie1 cookie2 => result +


+ +Returns true +if the cookies cookie1 +and cookie2 are equal. Two cookies are considered +to be equal if their names and paths are equal. + +
+ + + + + + +


[Function]
delete-old-cookies cookie-jar => cookie-jar +


+ +Removes all cookies from the cookie +jar cookie-jar which have either expired or +which don't have an expiry date. + +
+ + + +

Headers

+ +This section assembles a couple of convenience functions which can be +used to access information returned as the third +value (headers) +of HTTP-REQUEST. +

+Note that if the header +sends multiple headers +with the same name, these are comprised into one entry by +HTTP-REQUEST where the values +are separated by commas. + + + +


[Function]
header-value name headers => value +


+ +If headers is an alist of headers as returned by HTTP-REQUEST +and name is a keyword naming a header, this function returns the +corresponding value of this header (or NIL if it's not in +headers). +
+DRAKMA-USER 19 > (setq *header-stream* nil)
+NIL
+DRAKMA-USER 20 > (header-value :server
+                               (nth-value 2 (http-request "http://www.jalat.com/blogs/lisp?id=5")))
+"Hunchentoot 0.1.3 (TBNL 0.9.7)"
+
+ +
+ + + + + +


[Function]
split-tokens string => string-list +


+ +Splits the string string into a list of substrings separated +by commas and optional whitespace. Empty substrings are +ignored. +
+DRAKMA-USER 21 > (split-tokens "chunked, identity")
+("chunked" "identity")
+
+ +
+ + + + + + +


[Function]
read-tokens-and-parameters string &key value-required-p => list +


+ +Reads a comma-separated list +of tokens from the +string string. Each token can be followed by an +optional, semicolon-separated list +of attribute/value +pairs where the attributes are tokens followed by +a #\= character and a token or +a quoted string. +Returned is a list where each element is either a string (for a simple +token) or +a cons +of a string (the token) and +an alist +(the attribute/value pairs). If value-required-p +is NIL (the default is T), the value part +(including the #\= character) of each attribute/value +pair is optional. +

+An example of an HTTP header which uses a syntax which can be parsed +with this function is the 'Transfer-Encoding' header. +

+DRAKMA-USER 21 > (read-tokens-and-parameters "iso-8859-5, unicode-1-1;q=0.8")
+("iso-8859-5" ("unicode-1-1" ("q" . "0.8")))
+
+ +
+ + + + + +


[Function]
parameter-present-p name parameters => generalized-boolean +


+ +If parameters is +an alist +of parameters (i.e. of attribute/value pairs) as returned by, for +example, READ-TOKENS-AND-PARAMETERS and name is a string naming a +parameter, this function returns the full parameter (name and value) - +or NIL if it's not in parameters. +
+DRAKMA-USER 23 > (parameter-present-p "frob" '(("charset" . "latin-1") ("frob" . "quux")))
+("frob" . "quux")
+
+DRAKMA-USER 24 > (parameter-present-p "foo" '(("charset" . "latin-1") ("frob" . "quux")))
+NIL
+
+
+ + + + + + +


[Function]
parameter-value name parameters => value +


+ +If parameters is an alist of parameters (i.e. of attribute/value pairs) as returned by, for +example, READ-TOKENS-AND-PARAMETERS and name is a string naming a +parameter, this function returns the value of this parameter - or +NIL if it's not in parameters. +
+DRAKMA-USER 25 > (parameter-value "frob" '(("charset" . "latin-1") ("frob" . "quux")))
+"quux"
+
+DRAKMA-USER 26 > (parameter-value "foo" '(("charset" . "latin-1") ("frob" . "quux")))
+NIL
+
+
+ + + + + +


[Function]
get-content-type headers => type, subtype, parameters +


+ +Reads and parses a 'Content-Type' header and returns it as +three values - the type, the subtype, and an alist (possibly +empty) of name/value pairs for the optional parameters. headers +is supposed to be an alist of HTTP headers as returned by +HTTP-REQUEST. Returns NIL if there is no 'Content-Type' header amongst +headers. +
+DRAKMA-USER 27 > (get-content-type 
+                  (nth-value 2 (http-request "http://weitz.de/")))
+"text"
+"html"
+(("charset" . "iso-8859-1"))
+
+
+ + + + + + +
 

Acknowledgements

+ +Initial versions of Drakma used code +from ACL-COMPAT, +specifically the chunking code from Jochen Schmidt. (This has been replaced by Chunga.) +The API of +Drakma's HTTP-REQUEST was +inspired by John +Foderaro's DO-HTTP-REQUEST. +And greetings to Bob Hutchinson who +already anticipated this +library in 2005... :) + +

+This documentation was prepared with DOCUMENTATION-TEMPLATE. +

+

+$Header: /usr/local/cvsrep/drakma/doc/index.html,v 1.65 2007/04/07 18:21:03 edi Exp $ +

BACK TO MY HOMEPAGE + + + hunk ./drakma.asd 2 -;;; $Header: /usr/local/cvsrep/drakma/drakma.asd,v 1.32 2007/03/09 08:39:27 edi Exp $ +;;; $Header: /usr/local/cvsrep/drakma/drakma.asd,v 1.33 2007/04/07 18:21:01 edi Exp $ hunk ./drakma.asd 37 -(defvar *drakma-version-string* "0.6.2" +(defvar *drakma-version-string* "0.7.0" hunk ./drakma.asd 55 - #-:lispworks :trivial-sockets + #-:lispworks :usocket hunk ./request.lisp 2 -;;; $Header: /usr/local/cvsrep/drakma/request.lisp,v 1.41 2007/02/08 14:34:58 edi Exp $ +;;; $Header: /usr/local/cvsrep/drakma/request.lisp,v 1.42 2007/04/07 18:21:01 edi Exp $ hunk ./request.lisp 395 - (trivial-sockets:open-stream host port - :element-type 'octet)))) + (usocket:socket-stream + (usocket:socket-connect host port + :element-type 'octet))))) }