Majo🧙‍♀️

A memory profiler focusing on long-lived objects.

Motivation

I created this gem because I wanted to reduce the maximum memory usage of a Ruby program.

The existing memory profiler, memory_profiler gem, focuses on allocated and retained memory. This tool is useful to reduce allocations and retain memory, but it is not suitable for reducing the maximum memory usage.

This gem solves this problem by focusing on long-lived objects. Long-lived objects are objects that are not garbage collected for a long time. These objects are the main cause of the maximum memory usage.

NOTE: Performance

This gem slows down the target program because it hooks all object allocations and free. Also, it uses a lot of memory to store the object information. Therefore, it is not suitable in a production environment.

Installation

Install the gem and add to the application's Gemfile by executing:

$ bundle add majo --group development

If bundler is not being used to manage dependencies, install the gem by executing:

$ gem install majo

Usage

Wrap the code you want to profile with Majo.run. Then, call report method to display the result.

result = Majo.run do
  code_to_profile
end

result.report

Options

You can pass the following options to report method.

Name	Description	Default	Type
`out`	Output file or IO	`$stdout`	`IO`, `String` or an object having `to_path` method (such as `Pathname`)
`formatter`	The format of the result	`:color` or `monochrome`	`:color`, `:monochrome`, or `:csv`

For example:

result = Majo.run do
  code_to_profile
end

result.report(out: "majo-result.csv", formatter: :csv)

Result Example

The result contains only long-lived objects, which are collected by the major GC or retained.

The example is as follows:

Total 15055372 bytes (159976 objects)

Memory by file
-----------------------------------
10431760  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb
 1966348  /path/to/gems/rbs-3.5.1/lib/rbs/environment_loader.rb
  980344  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb
(snip)

Memory by location
-----------------------------------
10431760  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb:20
 1942812  /path/to/gems/rbs-3.5.1/lib/rbs/environment_loader.rb:159
  249920  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb:994
(snip)

Memory by class
-----------------------------------
4274028  String
3556632  RBS::Location
2197840  Array
(snip)

Objects by file
-----------------------------------
109126  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb
 20813  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb
 12236  /path/to/gems/rbs-3.5.1/lib/rbs/environment.rb
(snip)

Objects by location
-----------------------------------
109126  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb:20
  4458  /path/to/gems/rbs-3.5.1/lib/rbs/namespace.rb:24
  4132  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb:374
(snip)

Objects by class
-----------------------------------
52495  Array
22435  RBS::Location
11144  RBS::TypeName
(snip)

Retained Memory by file
-----------------------------------
7040  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/parser_aux.rb
 256  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/resolver/type_name_resolver.rb
(snip)

Retained Memory by location
-----------------------------------
7040  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/parser_aux.rb:20
 256  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/resolver/type_name_resolver.rb:22
(snip)

Retained Memory by class
-----------------------------------
6920  String
 256  Hash
 120  Symbol
(snip)

Retained Objects by file
-----------------------------------
160  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/parser_aux.rb
  1  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/resolver/type_name_resolver.rb
(snip)

Retained Objects by location
-----------------------------------
160  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/parser_aux.rb:20
  1  /path/to/gems/3.3.0/gems/rbs-3.5.1/lib/rbs/resolver/type_name_resolver.rb:22
(snip)

Retained Objects by class
-----------------------------------
157  String
  3  Symbol
  1  Hash
(snip)

The CSV format is as follows:

Object class path,Class path,Method ID,Singleton method,Path,Line,Alloc generation,Free generation,Memsize,Count
Hash,RBS::Parser,_parse_signature,true,/path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb,20,20,22,160,3
Hash,RBS::EnvironmentLoader,each_signature,false,/path/to/gems/rbs-3.5.1/lib/rbs/environment_loader.rb,159,20,22,160,1
Hash,RBS::Parser,_parse_signature,true,/path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb,20,21,23,160,1
(snip)

You can find the raw data in the CSV format. It is useful for further analysis. For example: A spreadsheet for Majo result

The columns are as follows:

Column name	Description
`Object class path`	The class name of the allocated object
`Class path`	The class name of the receiver of the method allocating the object
`Method ID`	The method name allocating the object
`Singleton method`	`true` if the method call is for a singleton method
`Path`	The file path of the method allocating the object
`Line`	The line number of the method allocating the object
`Alloc generation`	The GC generation number when the object is allocated
`Free generation`	The GC generation number when the object is freed. It's empty for retained objects.
`Memsize`	The memory size of the object in bytes
`Count`	Number of objects allocated with the same conditions

Name

The name "Majo" is a Japanese word 魔女 that means "witch". It is a wordplay on the word "major" in "major GC".

Thanks

This gem is inspired by memory_profiler gem.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/pocke/majo.