Memdump

Memdump is a set of (basic) tools to create and manipulate Ruby object dumps.

Since Ruby 2.1, ObjectSpace can be dumped in a JSON file that represents all allocated objects and their relationships. It is a gold mine of information if you want to understand why your application has that many objects and/or a memory leak.

Processing methods are available as a library, or using the memdump command-line tool. Just run memdump help for a summary of operations.

NOTE running memdump under jruby really reduces processing times... If you're using rbenv, just do

rbenv shell jruby-9.0.5.0

in the shell where you run the memdump commands.

Add this line to your application's Gemfile:

gem 'memdump'

And then execute:

$ bundle

Or install it yourself as:

$ gem install memdump

The memdump command-line tool can connect to a process where the excellent rbtrace has been required. Just start your Ruby application with -r rbtrace, e.g.

ruby -rrbtrace -S syskit run

and find out the process PID using e.g. top or ps (in the following, I assume that the PID is 1234)

Memory dumps are then created with

memdump dump 1234 /tmp/mydump

Since dump_all requires very long, the rbtrace client will return before the end of the dump with *** timed out waiting for eval response. Check your application's output for a line saying sendto(14): No such file or directory [detaching]

Additionally, you might want to enable allocation tracing, which adds to the dump the line/file of the point where the object got allocated but is also very costly from a performance point of view, do

memdump enable-allocation-trace 1234

It is sometimes more beneficial to do the dumps in specific places in your application, something the rbtrace method does not allow you to do. In this case, create memory dumps by calling ObjectSpace.dump_all

require 'objspace'
File.open('/path/to/dump/file', 'w') do |io|
  ObjectSpace.dump_all(output: io)
end

Allocation tracing is enabled with

require 'objspace'
ObjectSpace.trace_object_allocations_start

The first thing you will probably want to do is to run the replace-class command on the dump. It replaces the class attribute, which in the original dump is the reference to the class object, by the class name. This makes reading the dump a lot easier.

memdump replace-class /tmp/mydump

The most basic analysis is done by running stats, which outputs the object count by class. For memory leaks, the diff command allows you to output the part of the graph that involves new objects (removing the "old-and-not-referred-to-by-new")

Beyond, this analyzing the dump is best done through the interactive mode:

memdump interactive /tmp/mydump

will get you a pry shell in the context of the loaded MemoryDump object. Use the MemoryDump API to filter out what you need. If you're dealing with big dumps, it is usually a good idea to save them regularly with #dump.

One useful call to do at the beginning is #common_cleanup. It collapses the common collections (Array, Set, Hash) as well as internal bookkeeping objects (ICLASS, …). I usually run this, save the result and re-load the result (which is usually significantly smaller).

After, the usual process is to find out which non-standard classes are unexpectedly present in high numbers using stats, extract the objects from these classes with dump = objects_of_class('classname') and the subgraph that keeps them alive with roots_of(dump)

# Get the subgraph of all objects whose class name matches /Plan/ and export
# it to GML to process with Gephi (see below)
parent_dump, _ = roots_of(objects_of_class(/Plan/))
parent_dump.to_gml('plan-subgraph.gml')

Once you start filtering dumps, don't forget to simplify your life by cd'ing in the context of the newly filtered dumps

Beyond that, I usually go back and forth between the memory dump and gephi, a graph analysis application. to_gml allows to convert the memory dump into a graph format that gephi can import. From there, use gephi's layouting and filtering algorithms to get an idea of the shape of the dump. Note that you need to first get a graph smaller than a few 10k of objects before you can use gephi.

One powerful way to find out where memory is leaked is to look at objects that got allocated and find the interface between the long-term objects and these objects. memdump supports this by computing diffs.

If you mean to use dump diffs you MUST enable allocation tracing. Not doing so will make the diffs inaccurate, as memdump will not be able to recognize that some object addresses have been reused after a garbage collection.

Let's assume that we have a "before.json" and "after.json" dumps. Start an interactive shell loading before.

memdump interactive before.json

Then, in the shell, let's load the after dump

> after = MemDump::JSONDump.load('after.json')

The set of objects that are in after and before is given by #diff

d = diff(after)

We'll also add a special marker to the records in d so that we can easily colorize them differently in Gephi.

d = d.map { |r| r['in_after'] = 1; r }

One possibility is that there are only a few objects in the diff that are kept alive from before. These objects in turn keep alive a lot more objects (which cause the noticeable memory leak). What's interesting in this case is to visualize the interface, that is that set of objects.

In memdump, one computes it with the interface_with method, which computes the interface between the receiver and the argument. The receiver must contain the edges between itself and the argument, which means in our case that we must use after.

self_border, diff_border = after.interface_with(d)

In addition to computing the border, it computes the count of objects that are kept alive by each object in diff_border. Each record in diff_border has an attribute called keepalive_count that counts the amount of nodes in after that are reachable (i.e. kept alive by) it. It is usually a good idea to visualize the distribution of keepalive_count to see whether there's indeed only a few nodes, and whether some are keeping a lot more objects alive than others. Note that cycles that involve more than one "border node" will be counted multiple ones (so the sum of keepalive_count will be higher than d.size)

diff_border.size # is this much smaller than d.size ?
diff_border.each_record.map { |r| r['keepalive_count'] }.sort.reverse # are there some high counts at the top ?

From there, one needs to do a bunch of back-and-forth between memdump and Gephi. What I usually do is start by dumping the whole subgraph that contains the border and visualize. If I can't make any sense of it, I isolate the high-count elements in the border and visualize the related subgraph

full_subgraph = after.roots_of(diff_border)
full_subgraph.to_gml 'full.gml'
filtered_border = diff_border.find_all { |r| r['keepalive_count'] > 1000 }
filtered_subgraph = after.roots_of(filtered_border)
filtered_subgraph.to_gml 'filtered.gml'

Bug reports and pull requests are welcome on GitHub at https://github.com/doudou/memdump.

The gem is available as open source under the terms of the MIT License.