Vayacondios

Vayacondios is a server-client program designed to make it simple to collect and centralize information and metrics from a large number of disparate sources from multiple application domains.

Vayacondios has the following design goals:

The client is simple enough to use in a shell script and the server is performant enough to support ubiquitous use across a large installation with many clients.

• Decentralized -- Any client can dispatch stashes or events from anywhere
• Dynamic -- No data types or schemas need to be created in advance
• Ubiquitous -- Clients require minimal dependencies because the API is simple to use and access
• Simple -- Clients can write data in whatever way is natural for them
• Scalable -- Server and storage can be scaled horizontally to allow for ever-increasing loads
• Fast -- No client should have to worry that sending data to Vayacondios will affect its performance

The basic objects of Vayacondios are stash and the event:

• a stash is an "object", a "configuration", or "setting" designed to be shared among many services
• an event is a "fact", "measurement", or "metric" announced by an arbitrary service, possibly related to some stash

Stashes and events are each documents which can contain arbitrary JSON-serializable data: hashes/maps/dictionarys, arrays/lists, strings, numbers, floats, null, &c.

The client and server communicate over a RESTful, HTTP-based API which speaks JSON.

See also Coda Hale's metrics.

MongoDB is currently the only supported database. MongoDB is a natural choice because it exposes atomic query primitives which map very closely to the operations exposed by the Vayacondios API.

The Vayacondios server process is a Goliath web server which implements the Vayacondios API over HTTP using JSON.

A single server process can easily handle hundreds of client requests per second. Multiple Vayacondios servers can easily be deployed behind a load-balancer.

Each running server process reads and writes all its data in a single MongoDB database specified at runtime.

{
  "environment": "Jenkins CI v. 1.519",
  "project": {
    "name":    "website",
	"version": "0b4d99ded50a19e495d2472477bbb0784d8a18d8",
    "url":     "https://github.com/mycompany/website.git",
  },
  "build": {
    "time":   182,
	"status": "success"
  },
  "test": {
    "time":   97,
	"ran":    102,
	"passed": 102,
	"failed": 0
  }
}  

{
  "ip":     "74.210.29.117",
  "port":   22,
  "type":   "ssh",
  "reason": "blacklisted"
}

{
  "data_center": "Phoenix",
  "rack":        "14",
  "server":      "webserver-16",
  "cpu": {
    user:   3.17,
	nice:   0.01,
	system: 0.27,
	iowait: 0.18,
	steal:  0.00,
	idle:   96.38
  },
  "mem": {
    "total": 12304632,
	"used":  10335900,
	"free":  1968732
  },
  "net": {
    "out": 2.25,
	"in":  10.28,
  },
  "disk": {
    "write": 16.182,
	"read":  0.11
  }

{
  "projects": {
    {
	  "name": "website",
	  "url":  "https://github.com/mycompany/website.git",
	},
    {
	  "name": "client_tool",
	  "url":  "https://github.com/mycompany/client_tool.git",
	},
	...
  }
}

{
  "firewall": {
    "rules": [
	  {
	    "range":    "0.0.0.0",
		"port":     80,
		"protocol": "tcp"
      },
	  {
	    "range":    "10.0.0.0",
		"port";     22,
		"protocol": "ssh"
	  }
	]
  }
}

{
  "name":     "PHX",
  "location": "Phoenix, AZ",
  "servers": [
    "webserver-0",
	"webserver-1",
	"webserver-2",
	...
  ]
}

Vayacondios server depends on a database to store all its data. Currently, only MongoDB is supported: here are some installation instructions.

$ sudo gem install vayacondios-server

$ vcd-server --verbose --stdout

$ curl -X POST http://localhost:9000/v2/my_organization/event/some_topic -d '{"event": "data"}'

$ sudo gem install vayacondios-client

$ vcd announce 'some_topic' '{"event": "data"}'

---
host: vcd.example.com
port: 9000
organization: my_company

$ sudo gem install vayacondios-client

require 'vayacondios-client'
client = Vayacondios::HttpClient.new(organization: 'my_company')
client.announce('some_topic', foo: 'bar')

The Ruby client exposes several API requests as named methods (like announce above, which maps to a announce event API endpoint).

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  ...
  <repositories>
    ...
    <repository>
      <id>infochimps.releases</id>
      <name>Infochimps Internal Repository</name>
      <url>https://s3.amazonaws.com/artifacts.chimpy.us/maven-s3p/releases</url>
    </repository>
	...
  </repositories>
  ...
  <dependencies>
    ...
    <dependency>
      <groupId>com.infochimps</groupId>
      <artifactId>vayacondios</artifactId>
      <version>2.0.0</version>
    </dependency>
	...
  </dependencies>
  ...
</project>  

import com.infochimps.vayacondios.VayacondiosClient;
import com.infochimps.vayacondios.HTTPClient;

class public HelloVayacondios {
  public static void main(String[] args) throws Exception {
    VayacondiosClient client = new HTTPClient("my_organization");
	Map event = new HashMap();
	event.put("foo", "bar");
	client.announce("my_topic", event);
	Thread.sleep(50) // ensures async HTTP request finishes
    client.close();
  }
}

The Java client exposes several API requests as named methods (like announce above, which maps to a announce event API endpoint).

The version, organization, and type parameters are always required. Other parameters are required depending on the endpoint.

Vayacondios server only listens for a single value of the top-level version parameter (curently: v2). A frontend webserver (Apache, nginx, &c.) can be used to split traffic to backend Vayacondios servers running different versions of the Vayacondios API by routing based on this parameter.

The type parameter is fixed and defines the type of a Vayacondios request: event, stash, events, or stashes.

All other parameters are completely free for clients to specify under the following constraints:

• the organization parameter can only contain letters, digits, hyphens, and underscores and it must begin with a letter

• the topic parameter can only contain letters, digits, hyphens, underscores, and periods and it cannot start or end with a period

• the id parameter cannot contain the dollar sign or period

The type, organization, topic, and id parameter together constitue the vayacondios route.

The document is the request body sent to the server with a given request. Requests to Vayacondios should have JSON-encoded bodies but the body can be any JSON datatype: Hash, Array, String, Integer, Float, Boolean, or null.

The response is the JSON-encoded response body sent back to the client from the server. If an error occurred, in addition to the appropriate HTTP response code, the response will be a Hash containing the key error with a message detailing the error.

In the case of a record which is not found, the response may be empty but the HTTP response code will be 404.

In the case of a successful request, the response code will be 200 and the response body will the requested/written object.

All requests to announce a new event accept a Hash-like request body. Key/value pairs in this request body constitute the body of the event. The following parameters have special meaning:

The response body will contain a Hash that is the original request Hash but with the (possibly auto-generated) ID and timestamp included.

The response will contain the event Hash if found or will be empty if not.

The default behavior (which will occur with an empty request body) is to return the most recent 50 events on the given topic sorted in descending order by their timestamps.

Each key in the query body will be interpeted as a condition that the data of each event must match in order to be returned. Keys with periods are interpreted as nested fields. The following parameters have special meaning and can be used to adjust the time window, number of returned events, the sort behavior, and the fields within each event to return:

The response will be an Array of the matching events, possibly an empty Array if no events were found.

When setting the stash itself, your value must be Hash-like. When setting an ID within a stash, your value can have any datatype.

The response for setting a stash will be the (Hash-like) stash you just set. When setting an ID within a stash, the response will be of the same datatype as the request.

You can merge a value for a stash or one of the fields within a stash.

When merging the stash itself, your value must be Hash-like and will be merged on top of the existing (Hash-like) stash's value.

When merging one of the ID fields within the stash, your value can have any datatype and it will be intelligently merged:

• if your value is Hash-like and the existing value is Hash-like , your new value will be merged on top of the existing value
• if your value is Array-like and the existing value is Array-like , your new value will be concatenated to the end of the existing value
• if your value is String-like and the existing value is String-like , your new value will be concatenated to the end of the existing value
• if your value is Numeric-like and the existing value is Numeric-like , your new value will be added to the existing value

The response for merging a stash will be the Hash-like combination of your old and new value. The response for merging an ID within a stash will be of the same type as the request.

The response for retreiving a stash will be the Hash-like stash while the response for retreving an ID field within a stash will vary based on the datatype of that value.

The response for deleting a stash or an ID within a stash will be a Hash naming the topic (and ID if given in the request) deleted.

The default behavior (which will occur with an empty request body) is to return 50 stashes in sorted in ascending order by their topic.

Each key in the query body will be interpreted as a condition that the data of each stash must match in order to be returned. Keys with periods are interpreted as nested fields. The following parameters have special meaning and can be used to adjust the number of returned stashes, the fields within each stash, and the the sort behavior.

The response will be an Array of the matching stashes, possibly an empty Array if no events were found.

There are two related methods that let you update multiple stashes in place according to the same rule.

Each of these methods accepts a Hash request body that is supposed to have the following parameters:

The methods differ in the way they process the update. The POST method will always replace the existing values of the named fields (top-level or nested) in the update Hash with their new, updated values. This is useful when you want to explicitly set a property to some fixed value for all stashes which match some criteria.

The PUT method (like the PUT /v2/:organization/stash/:topic method) tries to be more clever and attempts to merge the update into the record according to the following rules

• if your value is Numeric-like, your new value will be added to the existing value
• if your value is Array-like, your new value will be concatenated to the end of the existing value
• if your value is Hash-like, it will replace the existing value (Note:: this is different than the behavior of PUT /v2:organization/stash/:stopic.)
• if your value is String-like, it will replace the existing value (Note:: this is different than the behavior of PUT /v2:organization/stash/:stopic.)

Notice that the behavior for Hashes and Strings isn't as nice as it is for PUT /v2:organization/stash/:stopic. Still, this method is useful for when you want to increment some value across all stashes which match some criteria.

For both of these methods, the query Hash also contains parameters that are very similar to when searching for a stash:

Other fields in the query Hash are interpreted as requirements on the data in the stash itself.

The following method can be used to delete multiple stashes which all match some criteria:

The following request parameters define the query which is used to match stashes to be deleted:

Other fields in the query Hash are interpreted as requirements on the data in the stash itself in order for it to match and be deleted.

Copyright (c) 2011 - 2013 Infochimps. See LICENSE.md for further details.