CL-MEMCACHED - Common Lisp interface to the memcached object caching system.


 

Abstract

CL-MEMCACHED is a library to interface with the memcached object caching system.

What is Memcached?? According to the home page :

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Danga Interactive developed memcached to enhance the speed of LiveJournal.com, a site which was already doing 20 million+ dynamic page views per day for 1 million users with a bunch of webservers and a bunch of database servers. memcached dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss.

CL-MEMCACHED implements most of the memcached protocol. The code has been tested on Allegro CL and does not work on other Lisp's right now. See file compat.lisp to help.

We have used memcached (1.1.2) in production for over 20 months and have found it to give excellent performance and good stability. The CL-MEMCACHED has evolved over this period of time from a hack to it's current state. Our memcached servers have been up for over 60 days at a time having served over a terabyte of data to the network in this period.

Here are some sample performance statistics of CL-MEMCACHED and other memcached clients :
clientlang implementation10,000 writes
1K data (in msec)
10,000 reads
1K data (in msec)
10,000 writes
10K data (in msec)
10,000 reads
10K data (in msec)
cl-memcachedAllegro 8.0 (AMD64)9508302,1301,330
memcached-client-1.2.0ruby 1.8.57278741,1291,296
python-memcached-1.36python 2.5.18929511,0921,259
php-memcached-2.1.2php 4.3.9507513400,000660

The code comes with a BSD-style license so you can basically do with it whatever you want.

Download shortcut: cl-memcached-latest.tar.gz.

 

Lists


The devel mailing list is cl-memcached-devel.
The announce mailing list is cl-memcached-announce.
 

Example Usage

quick start

CL-USER> (asdf:oos 'asdf:load-op :cl-memcached)

CL-USER> (setf *my-cache* (cl-memcached:mc-make-memcache-instance :ip "127.0.0.1" :name "My test cache"))
#<CL-MEMCACHED:MEMCACHE My test cache on 127.0.0.1:11211 SIZE:64Mb>

CL-USER> (cl-memcached:mc-store "test-key" "This is Test-DATA" :memcache *my-cache* :use-pool t)
"STORED"

CL-USER> (cl-memcached:mc-get+ "test-key" :memcache *my-cache* :use-pool t)
"This is Test-DATA"

CL-USER> (cl-memcached:mc-get '("test-key") :memcache *my-cache* :use-pool t)
(("test-key"
  #(84 104 105 115 32 105 115 32 84 101 115 116 45 68 65 84 65)))

CL-USER> (cl-memcached:mc-get '("test-key") :memcache *my-cache* :use-pool t :is-string t)
(("test-key" "This is Test-DATA"))

CL-USER> (cl-memcached:mc-store "test-key-2" "This is Test-DATA Again" :memcache *my-cache* :use-pool t)
"STORED"

CL-USER> (cl-memcached:mc-get '("test-key" "test-key-2") :memcache *my-cache* :use-pool t :is-string t)
(("test-key" "This is Test-DATA")
 ("test-key-2" "This is Test-DATA Again"))


 

Contents

  1. Download
  2. The CL-MEMCACHED dictionary
    1. *memcache*
    2. *use-pool*
    3. *pool-get-trys?*
    4. mc-decr
    5. mc-del
    6. mc-get
    7. mc-get+
    8. mc-incr
    9. mc-make-memcache-instance
    10. mc-pool-init
    11. mc-server-check
    12. mc-stats
    13. mc-store
    14. memcache
  3. Acknowledgements

 

Download

CL-MEMCACHED together with this documentation can be downloaded from cl-memcached-latest.tar.gz. The current version is 0.4.1. (I will be setting up a SVN repo soon)
 

The CL-MEMCACHED dictionary


[Special variable]
*memcache*


We can set the current memcache instance to this if there is only one in use.


[Special variable]
*use-pool*


This controls if we use the connection pool by default. One can set it at each call level, but it is also
possible to set this global policy.

Default value for the USE-POOL is nil, which means a new connection is make every request.


[Special variable]
*pool-get-trys?*


This controls the policy for the fetching connectors from the pool.  There are two approaches :
a) where we throw an error if pool is empty
b) where we sleep an try again to see if one is available.

The default value is nil which is the a) approach.


[Function]
mc-store key data &key memcache command timeout use-pool => result


Stores data in the memcached server.
key - key by which the data is stored. this is of type SIMPLE-STRING
data - data to be stored into the cache. data is a sequence of type (UNSIGNED-BYTE 8)
length - size of data
memcache - The instance of class memcache which represnts the memcached we want to use.
command - The storage command we want to use.  There are 3 available : set, add & replace.
timeout - The time in seconds when this data expires.  0 is never expire.


[Function]
mc-get keys-list &key memcache use-pool is-string => result


Retrive value for key from memcached server.
keys-list - is a list of the keys, seperated by whitespace, by which data is stored in memcached
memcache - The instance of class memcache which represnts the memcached we want to use.

Returns a list of lists where each list has two elements key and value
key - is of type SIMPLE-STRING
value is of type (UNSIGNED-BYTE 8)


[Function]
mc-get+ key-or-list-of-keys &key memcache use-pool => result


To be used for non-binary data only. If one key is given returns the response in string format


[Function]
mc-decr key &key memcache value use-pool => result


Implements the DECR command. Decrements the value of a key. Please read memcached documentation for more information


[Function]
mc-del key &key memcache time use-pool => result


Deletes a particular 'key' and it's associated data from the memcached server


[Function]
mc-incr key &key memcache value use-pool => result


Implements the INCR command. Increments the value of a key. Please read memcached documentation for more information


[Function]
mc-make-memcache-instance &key ip port name pool-size => result


Creates an instance of class MEMCACHE which represents a memcached server


[Function]
mc-pool-init &key memcache => result


Cleans up the pool for this particular instance of memcache & reinits it with POOL-SIZE number of objects required by this pool


[Function]
mc-server-check &key memcache => result


Performs some basic tests on the Memcache instance and outputs a status string


[Function]
mc-stats &key memcache use-pool => result


Returns a struct of type memcache-stats which contains internal statistics from the memcached server instance. Please refer to documentation of memcache-stats for detailed information about each slot.


[Structure]
memcache-stats


The structure which holds the statistics from the memcached server. The fields are :
field-name                 accessor-function                 documentation
----------                 -----------------                 -------------
pid                        mc-stats-pid                      Process id of this server process
uptime                     mc-stats-uptime                   Number of seconds this server has been running
time                       mc-stats-time                     current UNIX time according to the server
version                    mc-stats-version                  Version string of this server
rusage-user                mc-stats-rusage-user              Accumulated user time for this process
rusage-system              mc-stats-rusage-system            Accumulated system time for this process
curr-items                 mc-stats-curr-items               Current number of items stored by the server
total-items                mc-stats-total-items              Total number of items stored by this server ever since it started
bytes                      mc-stats-bytes                    Current number of bytes used by this server to store items
curr-connections           mc-stats-curr-connections         Number of open connections
total-connections          mc-stats-total-connections        Total number of connections opened since the server started running
connection-structures      mc-stats-connection-structures    Number of connection structures allocated by the server
cmd-get                    mc-stats-cmd-get                  Cumulative number of retrieval requests
cmd-set                    mc-stats-cmd-set                  Cumulative number of storage requests
get-hits                   mc-stats-get-hits                 Number of keys that have been requested and found present
get-misses                 mc-stats-get-misses               Number of items that have been requested and not found
evictions                  mc-stats-evictions                Number of items removed from cache because they passed their expiration time
bytes-read                 mc-stats-bytes-read               Total number of bytes read by this server from network
bytes-written              mc-stats-bytes-written            Total number of bytes sent by this server to network
limit-maxbytes             mc-stats-limit-maxbytes           Number of bytes this server is allowed to use for storage.


[Standard class]
memcache


This class represents an instance of the Memcached server
(defclass memcache ()
  ((name
    :initarg :name
    :reader name
    :type simple-string
    :documentation "Name of this Memcache instance")
   (ip 
    :initarg :ip
    :initform "127.0.0.1"
    :accessor ip
    :type simple-string
    :documentation "The IP address of the Memcached server this instance represents")
   (port 
    :initarg :port
    :initform 11211
    :accessor port
    :type fixnum
    :documentation "The port on which the Memcached server this instance represents runs")
   (memcached-server-storage-size 
    :initform 0
    :reader memcached-server-storage-size
    :type fixnum
    :documentation "Memory allocated to the Memcached Server")
   (pool-size
    :initarg :pool-size
    :initform 2
    :reader pool-size)
   (pool
    :reader pool))
  (:documentation "This class represents an instance of the Memcached server"))

 

Known Issues

The pooling functionality is still experimental. This is mainly because strategies to deal with network errors are not in place.
 

TODO


- Add facility to created a replicated memcached pair.
- Support for a memcached cluster (distributedness)

People

Abhijit 'quasi' Rao

Chaitanya Gupta
 

Acknowledgements

Thanks to Mr. Hrush Bhatt of Cleartrip for allowing us to make this library available under a BSD licence.

This documentation was prepared with the help of DOCUMENTATION-TEMPLATE.