* 0XBAD broadcasty distributed shared memory key/value cache the whole thing is horribly simple, using FNV32 hashing to build a table (without collision detection - so if 2 keys collide they are overwritten every time) [box A] - my $v = $cache->find($key) || $cache->store_and_broadcast($key,slow_subroutine(),0,12345); [box B] - daemon receives the broadcast message; adds it into the local cache [box C] - daemon receives the broadcast message; adds it into the local cache [box D] - udp is lost in this case, when a request for $key hits [box D] it will do the same $cache->store_and_broadcast() and rebroadcast the item. * install ruby: gem install xbad perl: cd perl && perl Makefile.PL && make && make test && make install * usage my $CACHE = BAD->new(0xbeef,16,128) creates new shared memory segment with id 0xBEEF the segment size is 2^16 * 128 bytes, and it can hold at most 2^16 items my $value = $CACHE->find($key) - looks up my $value = $CACHE->store($key,$x_value,$expire_after); my $value = $CACHE->store_and_broadcast($key,$x_value,$expire_after,$broadcast_port); if $expire_after is 0, the item never expires ruby: b = XBAD.new(0xBAD,16,128) b.store(key,value,expire_after) value = b.find(key) * broadcasting store_and_broadcast sends udp broadcast to the specified port * fixed size every item in the cache is with size sp->item_size (specified on pool creation) and the shared memory segment is created when you specify exactly how many items between 2^16 and 2^32 you want to have in the pool. my $b = BAD->new(0xBEEF,17,128); # will create 2^17 items with size 128 bytes this is very ineficient, but for caching small-ish items it works fine. * speed perl: (key: aaaaaaaaaaaaaaaabbbbbbbbbbbbbbb value 'a' x 100) native hash store: 4644160.73/s 0xBAD store: 3062807.05/s memcached store: 24514.01/s native hash get: 3476691.62/s 0xBAD get: 2153601.64/s memcached get: 25216.77/s * native hash - perl's $hash{$key} = $value and $value = $hash{$key} ruby: hash set 3173242.4 i/s - 15841482 in 4.993171s xbad set 3760115.4 i/s - 18829228 in 5.008079s hash get 4008048.8 i/s - 20056464 in 5.004359s xbad get 1844151.9 i/s - 8363250 in 5.010949s hash mis 4262632.4 i/s - 21317955 in 5.001340s xbad mis 4441230.5 i/s - 22199103 in 4.999691s memcache set 13164.1 i/s - 66689 in 5.070756s memcache get 14114.6 i/s - 70788 in 5.026147s memcache cset 12767.4 i/s - 64501 in 5.076763s memcache cget 13474.4 i/s - 68202 in 5.082600s hash clone set 437421.6 i/s - 2196828 in 5.031107s hash clone get 446848.6 i/s - 2230452 in 4.996158s xbad set-encoded 794074.4 i/s - 3953088 in 5.006689s xbad get-encoded 372012.9 i/s - 1855488 in 5.035192s xbad set-encoded-sn 558399.0 i/s - 2798724 in 5.023483s xbad get-encoded-sn 372813.0 i/s - 1858272 in 5.028326s * hash- is ruby's native hash hash[key] = value and value = hash[key] * hash clone is hash[key] = value.clone and value = hash[key].clone xbad - encoded is Sereal (https://github.com/Sereal/Sereal) encoded xbad encoded-sn is Sereal encoded and snappy compressed * locking everything is protected by simple gcc atomic builtins worst case is somebody locks an item and goes away (interrupted by signal or something)
Project
xbad
xbad - shared memory key/value cache using shared memory and udp broadcasts for distribution
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
Development
Dependencies
Development
>= 0
Project Readme