0.01
No commit activity in last 3 years
No release in over 3 years
Vector embedding of strings, booleans, numerics, and arrays into LIBSVM / LIBLINEAR format.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0
>= 0
>= 0

Runtime

 Project Readme

VectorEmbed

Vector embedding of strings, booleans, numerics, and arrays into LIBSVM / LIBLINEAR format.

Inspired by Sally, except VectorEmbed is meant to handle categorical and continuous data at the same time.

csv2libsvm

$ cat houses.csv 
label,household_income,zip_code_id,year
+1,35893,53703,1904
-1,43708,53711,1977
+1,103214,53719,NULL
-1,49250,53704,1950

$ csv2libsvm houses.csv 
1 1243483:35893 6439848:53703 8227451:1904
-1 1243483:43708 6439848:53711 8227451:1977
1 1243483:103214 6439848:53719 8227451:0
-1 1243483:49250 6439848:53704 8227451:1950

Usage

Create a VectorEmbed instance, which auto-detects and then remembers what kind of data goes into each feature:

>> require 'vector_embed'
=> true
>> v = VectorEmbed.new
=> #<VectorEmbed:0x007fd605815208 [...]>

Output a line with a label and arbitrary features:

>> label = 1
=> 1
>> features = { color: 'red', year: 1995, weight: 5.4e9 }
=> {:color=>"red", :year=>1995, :weight=>5400000000.0}
>> v.line(label, features)
=> "1 1997960:1 5556418:5400000000.0 8227451:1995"

Output another line:

>> label = 0
=> 0
>> features = { color: 'blue', year: 1821, weight: 3.3 }
=> {:color=>"blue", :year=>1821, :weight=>3.3}
>> v.line(label, features)
=> "0 1089740:1 5556418:3.3 8227451:1821"

Note that color: 'red' and color: 'blue' are being translated into categories:

1997960:1 # murmur3("color\x00red"):1
1089740:1 # murmur3("color\x00blue"):1

A similar thing happens with true/false:

>> v.line(1, yes: true, no: false)
=> "1 1559987:1 3324244:1"

i.e.

1559987:1 # murmur3("yes\x00true"):1
3324244:1 # murmur3("no\x00false"):1

N-grams

Currently uses same parameter names as Sally.

Word ngrams

>> v = VectorEmbed.new ngram_len: 1, ngram_delim: /\s+/, stop_words: %w{the and or}
=> #<VectorEmbed:0x007fd6033b77f8 [...]>
>> v.line(1, notes: 'the quick brown fox')
=> "1 1512788:1 3426202:1 5079692:1"

You get the idea: ("the" has been filtered out by stop words)

1512788:1 # murmur3("notes\x00quick"):1
3426202:1 # murmur3("notes\x00brown"):1
5079692:1 # murmur3("notes\x00fox"):1

Byte n-grams

>> v = VectorEmbed.new ngram_len: 2, ngram_delim: ''
=> #<VectorEmbed:0x007fd60337ea20 [...]>
>> v.line(1, notes: 'foobar')
=> "1 2148745:1 2878919:1 3600333:1 3621715:1 5885921:1"

So therefore:

2148745:1 # murmur3("notes\x00fo"):1
2878919:1 # murmur3("notes\x00oo"):1
3600333:1 # murmur3("notes\x00ob"):1
3621715:1 # murmur3("notes\x00ba"):1
5885921:1 # murmur3("notes\x00ar"):1

Debugging

VectorEmbed tries to do the right thing, but if it's not, try turning on debugging:

>> v = VectorEmbed.new
=> #<VectorEmbed:0x007fd6034020a0 [...]>
>> v.logger.level = Logger::DEBUG
=> 0
>> v.line(1, '3' => 7, foo: 'bar', truthy: false, nullity: nil)
D, [2013-02-20T16:55:00.139299 #21595] DEBUG -- : Interpreting "3" as Number given first value 7
D, [2013-02-20T16:55:00.139561 #21595] DEBUG -- : Interpreting :foo as Phrase given first value "bar"
D, [2013-02-20T16:55:00.139671 #21595] DEBUG -- : Interpreting :truthy as Boolean given first value false
D, [2013-02-20T16:55:00.139755 #21595] DEBUG -- : Interpreting :nullity as Boolean given first value nil
D, [2013-02-20T16:55:00.139872 #21595] DEBUG -- : Interpreting "label" as Number given first value 1
=> "1 2647413:7 4091306:1 7123386:1 9259635:1"

One thing it doesn't like: (assuming you have already performed the lines above)

>> v.line(1, '3' => 'bar')
ArgumentError: Can't embed "bar" in number feature "3".

It's saying that, given you first passed it 7, it thought "3" was a feature that held numbers.

Gotchas

  • Following Sally, it only uses the first 22 bits of the murmur hash for feature indices... more and LIBSVM seems to choke.
  • Stop words are currently filtered out of feature indices... probably shouldn't be.

Copyright

Copyright 2013 Seamus Abshere