README
Name
grntest
Description
Grntest is a testing framework for Groonga. You can write a test for Groonga by writing Groonga commands and expected result.
Install
% gem install grntest
Basic usage
Write a test script that extension is .test
. Here is a sample test
script select.test
:
table_create Users TABLE_HASH_KEY ShortText
load --table Users
[
{"_key": "Alice"},
{"_key": "Bob"}
]
select Users --query '_key:Alice'
Run grntest
with select.test
as command line argument:
% grntest select.test
N
================================================================================
.
select 0.3866s [not checked]
================================================================================
table_create Users TABLE_HASH_KEY ShortText
[[0,0.0,0.0],true]
load --table Users
[
{"_key": "Alice"},
{"_key": "Bob"}
]
[[0,0.0,0.0],2]
select Users --query '_key:Alice'
[[0,0.0,0.0],[[[1],[["_id","UInt32"],["_key","ShortText"]],[1,"Alice"]]]]
================================================================================
tests/sec | tests | passes | failures | leaked | !checked |
2.57 | 1 | 0 | 0 | 0 | 1 |
0% passed in 0.3885s.
It generates select.actual
file that contains actual result. If it
is expected result, rename it to select.expected
:
% mv select.actual select.expected
Run grntest
again:
% grntest select.test
.
tests/sec | tests | passes | failures | leaked | !checked |
5.76 | 1 | 1 | 0 | 0 | 0 |
100% passed in 0.1736s.
It compares actual result and content of select.expected
and
reporots compared result. If they are the same contnet, grntest
reports success. If they are not the same content, grntest
reports
failure and show diff of them.
Change --query '_key:Alice'
to --query '_key:Bob
' in
select.test
:
table_create Users TABLE_HASH_KEY ShortText
load --table Users
[
{"_key": "Alice"},
{"_key": "Bob"}
]
select Users --query '_key:Bob'
Run grntest
again:
% grntest select.test
F
================================================================================
.
select 0.1767s [failed]
================================================================================
--- (expected)
+++ (actual)
@@ -6,5 +6,5 @@
{"_key": "Bob"}
]
[[0,0.0,0.0],2]
-select Users --query '_key:Alice'
-[[0,0.0,0.0],[[[1],[["_id","UInt32"],["_key","ShortText"]],[1,"Alice"]]]]
+select Users --query '_key:Bob'
+[[0,0.0,0.0],[[[1],[["_id","UInt32"],["_key","ShortText"]],[2,"Bob"]]]]
================================================================================
tests/sec | tests | passes | failures | leaked | !checked |
4.65 | 1 | 0 | 1 | 0 | 0 |
0% passed in 0.2153s.
It says the expected result that is read from select.expected
and
the actual result are not same. And the difference of them is shown in
unified diff format. It is helpful to debug the test.
select.reject
file is generated on failure. It contains the actual
result. If the actual result is the expected result, rename it to
select.expected
.
% mv select.reject select.expected
Run grntest
again:
% grntest select.test
.
tests/sec | tests | passes | failures | leaked | !checked |
6.20 | 1 | 1 | 0 | 0 | 0 |
100% passed in 0.1613s.
The test is succeeded again.
Advanced usage
There are more useful features. They are not needed for normal users but they are very useful for advanced users.
There are advanced features:
- Comment
- Continuation line
- Directives
Comment
Groonga supports comment line by #
.
Example:
# This line is comment line.
select Users
Grntest also supports the syntax. You can use #
as comment mark.
Continuation line
You can break a long line by escaping new line with \
.
Example:
select Users \
--match_columns name \
--query Ken
The command is processed as the following command:
select Users --match_columns name --query Ken
You can make your test readable with this feature.
Groogna doesn't support this feature.
Directives
Grntest supports directives that control grntest behavior.
Here is directive syntax:
#@NAME [ARGUMENTS...]
Here are available NAME
s:
disable-logging
enable-logging
suggest-create-dataset
include
copy-path
long-timeout
on-error
omit
add-important-log-levels
remove-important-log-levels
sleep
collect-query-log
ARGUMENTS...
are depends on directive. A directive doesn't require
any arguments but a directive requires arguments.
disable-logging
Usage:
#@disable-logging
It disables logging executed command and executed result until
enable-logging
directive is used. It is useful for executing
commands that isn't important for test.
Example:
#@disable-logging
load --table Users
[
{"_key": "User1"},
{"_key": "..."},
{"_key": "User9999999"}
]
#@enable-logging
select Users --query _key:User29
See also: enable-logging
enable-logging
Usage:
#@enable-logging
It enables logging that is disabled by disable-logging
directive.
See also: disable-logging
suggest-create-dataset
Usage:
#@suggest-create-dataset DATASET_NAME
It creates dataset DATASET_NAME
for suggest feature. It is useful
for testing suggest feature.
Example:
#@suggest-create-dataset rurema
load --table event_rurema --each 'suggest_preparer(_id, type, item, sequence, time, pair_rurema)'
[
["sequence", "time", "item", "type"],
["21e80fd5e5bc2126469db1c927b7a48fb3353dd9",1312134098.0,"f",null],
["21e80fd5e5bc2126469db1c927b7a48fb3353dd9",1312134105.0,"er",null],
["21e80fd5e5bc2126469db1c927b7a48fb3353dd9",1312134106.0,"erb",null],
["21e80fd5e5bc2126469db1c927b7a48fb3353dd9",1312134107.0,"erb","submit"]
]
See also: --groonga-suggest-create-dataset
option
include
Usage:
#@include SUB_TEST_FILE_PATH
It includes SUB_TEST_FILE_PATH
content. It is useful for sharing
commands by many tests.
You can use include
in included file. It means that an included file
is processed in the same way as a test file.
If SUB_TEST_FILE_PATH
is relative path, SUB_TEST_FILE_PATH
is
found from base directory. Base directory can be specified by
--base-directory
option.
Example:
init.grn:
#@disable-logging
#@include ddl.grn
#@include data.grn
#@enable-logging
ddl.grn:
table_create Users TABLE_HASH_KEY ShortText
data.grn:
load --table Users
[
["_key"],
["Alice"],
["Bob"]
]
user.test:
#@include init.grn
select Users --query _key:Alice
See also: --base-directory
option
copy-path
Usage:
#@copy-path SOURCE DESTINATION
It copies a path from SOURCE
to DESTINATION
. You can use it for
both file and directory. It is useful for using fixture data.
You can use #{db_path}
in SOURCE
and DESTINATION
. #{db_path}
is expanded to the path of the database.
Example:
#@copy-path fixture/query_expander/tsv/japanese_synonyms.tsv tmp/synonyms.tsv
register "query_expanders/tsv"
long-timeout
Usage:
#@long-timeout TIMEOUT
It specifies a timeout for commands that may take long time.
TIMEOUT
must be a number or default
. If you specify default
as
TIMEOUT
, the default timeout is used. The default timeout is 180
seconds.
Here are the commands that may take long time:
column_create
register
Example:
# Wait 300 seconds until commands that may take long time output their results
#@long-timeout 300
column_create Lexicon users_name COLUMN_INDEX Users name
# Reset custom timeout for commands that may take long time.
#@long-timeout default
on-error
Usage:
#@on-error ACTION
It specifies a action on error. ACTION
must be default
or
omit
. If you specify default
as ACTION
, the running test is
continued. If you specify omit
as ACTION
, the running test is
aborted and the running test is marked as omitted.
Example:
# Omit this test if TokenKyTea tokenizer isn't available.
#@on-error omit
register tokenizers/kytea
# Don't omit this test when any error is occurred in the following commands.
#@on-error default
omit
Usage:
#@omit REASON
Omit the test with REASON
.
Example:
# Omit this test until the feature is implemented.
#@omit "This feature is not implemented yet."
new_excelent_command
add-important-log-levels
Usage:
#@add-important-log-levels LEVEL_1 LEVEL_2 ...
Add LEVEL_N
to important log level list. grntest outputs log
messages when their log level is included in important log level
list.
The default important log level list has the following log levels:
emergency
alert
critical
error
warning
notice
If you want to test log messages for info
, debug
or dump
log
levels, you need to add the log level to important log level list.
Example:
# Enable logs for debug level
log_level --level debug
# Collect debug level log messages
#@add-important-log-levels debug
log_put --level debug --message "This is a message"
# The following messages is collected by grntest
#|d| This is a message
remove-important-log-levels
Usage:
#@remove-important-log-levels LEVEL_1 LEVEL_2 ...
Remove LEVEL_N
from important log level list. grntest outputs log
messages when their log level is included in important log level
list.
The default important log level list has the following log levels:
emergency
alert
critical
error
warning
notice
You can remove them from important log level list. You can only remove
log levels added by add-important-log-levels
.
Example:
# Enable logs for debug level
log_level --level debug
# Collect debug level log messages
#@add-important-log-levels debug
log_put --level debug --message "This is a message"
# The following messages is collected by grntest
#|d| This is a message
# Disable collecting debug level log messages
#@remove-important-log-levels debug
log_put --level debug --message "This is a message"
# No message is collected by grntest
sleep
Usage:
#@sleep SECOND
It stops execution while the specified seconds.
Example:
load --table Users
[
{"_key": "User1"}
]
#@sleep 1.5
# Run select after 1.5 sec.
select Users --query _key:User1
collect-query-log
Usage:
#@collect-query-log [true|false]
It collects messages from query log and prints them.
Example:
#@collect-query-log true
select Users --query _key:User1
#@collect-query-log false
Options
Grntest has many options. You don't need to specify many of them because they use suitable default values.
This section describes some important options. You can see all options
by grntest --help
. You will find many usuful features from it.
--test
Usage:
% grntest --test TEST_NAME ...
% grntest --test /TEST_NAME_REGEXP/ ...
% grntest --test TEST_NAME1 --test TEST_NAME2 ...
% grntest --test /TEST_NAME_REGEXP1/ --test /TEST_NAME_REGEXP2/ ...
--test
option specifies tests that should be ran. It is useful when
you are interested in only one test.
For example, the following command line runs only vector-geo-point
test.
% grntest --test vector-geo-point ...
You can use --test
option multiple times to run only multiple
interested tests.
For example, the following command line runs only vector-geo-point
and hash
tests.
% grntest --test vector-geo-point --test hash ...
You can use regular expression to select tests by /.../
syntax.
For example, the following command line runs tests that have geo
in
its name.
% grntest --test /geo/ ...
You can also use multiple --test options with regular expression.
See also: --exclude-test
option
--exclude-test
--test-suite
--exclude-test-suite
--gdb
--keep-database
--n-workers
--n-workers
option is very useful. You can run many test scripts at
once. Tests are finished quickly. You should specify one or more
directories that contain many test scripts. If you have only a test
script, --n-workers
is not effective.
Here is a sample command line to use --n-workers
:
% grntest --n-workers 4 test/function/suite/suggest
tests/sec | tests | passes | failures | leaked | !checked |
[0] [finished]
9.95 | 5 | 5 | 0 | 0 | 0 |
[1] [finished]
9.22 | 4 | 4 | 0 | 0 | 0 |
[2] [finished]
9.11 | 4 | 4 | 0 | 0 | 0 |
[3] [finished]
10.10 | 5 | 5 | 0 | 0 | 0 |
|-----------------------------------------------------------------------| [100%]
tests/sec | tests | passes | failures | leaked | !checked |
34.61 | 18 | 18 | 0 | 0 | 0 |
100% passed in 0.5201s.
--base-directory
--database=PATH
Usage:
% grntest --database /PATH/TO/EXISTING/DATABASE ...
--database
option specifies an existing database that is used for
test. Normally, grntest uses a new empty database for test.
--database
option is very useful for testing against existing
database.
--groonga
--groonga-httpd
--groonga-suggest-create-dataset
--interface
--output-type
--testee
Examples
See test/command/ directory in Groonga's source. It has many test scripts and uses many useful features. They will help you.
Dependencies
- Ruby
- msgpack gem
Mailing list
- English: groonga-talk@lists.sourceforge.net
- Japanese: groonga-dev@lists.sourceforge.jp
Thanks
- ...
Authors
- Kouhei Sutou <kou@clear-code.com>
- Haruka Yoshihara <yoshihara@clear-code.com>
License
GPLv3 or later. See doc/text/gpl-3.0.txt for details.
(Kouhei Sutou has a right to change the license including contributed patches.)