Gistflow

Benchmarking with Ruby by makaroni4

A great technique to improve your code is using benchmarking – comparison between different solutions leads to faster and better code.

Examples under the cut.

#ruby already has Benchmark module which can be used for measuring time of running of your code. For example lets compare string concatenation methods += and <<:

require 'benchmark'

Benchmark.bm do |r|
  N = 100000

  r.report("+= ") do
    s = ""
    N.times { s += "1" }
  end

  r.report("<< ") do
    s = ""
    N.times { s << "1" }
  end
end

# =>         user           system        total         real
# => +=   1.060000   0.340000   1.400000 (  1.405588)
# => <<   0.030000   0.000000   0.030000 (  0.025614)

This case is great and you can see the huge difference between using += and << methods. But benchmark timing approach is not scientific – time depends on many params and the results will be different for each run. Good approach is to run your benchmark 'experiment' several times and than calculate average.

Take a look at #gem benchmark_suite, it is a great for benchmarking. Basically it provides extension for standart benchmark. So lets look at previous example implemented with benchmark_suite:

require 'benchmark'
require 'benchmark/ips'

Benchmark.ips do |r|
  N = 100000

  r.report("+= ") do
    s = ""
    N.times { s += "." }
  end

  r.report("<< ") do
    s = ""
    N.times { s << "." }
  end
end

# => Calculating -------------------------------------
# =>                 +=          1 i/100ms
# =>                 <<          3 i/100ms
# =>-------------------------------------------------
# =>                 +=         0.2 (±0.0%) i/s -          2 in   8.508143s
# =>                 <<        33.9 (±3.0%) i/s -        171 in   5.054061s

Also we can run benchmark without any N specified because report blocks will be executed multiple times:

require 'benchmark'
require 'benchmark/ips'

Benchmark.ips do |r|  
  r.report("+ ") do
    42 + 42
  end

  r.report("* ") do
    42 * 42
  end
end

# =>Calculating -------------------------------------
# =>                  +      92249 i/100ms
# =>                  *      91950 i/100ms
# =>-------------------------------------------------
# =>                  +   6852082.4 (±6.8%) i/s -   34039881 in   4.998483s
# =>                  *   6280292.3 (±5.5%) i/s -   31263000 in   4.996057s

Obviously benchmark_suite takes longer time but the results are more precise. Use this technique to test your class methods, algorithms and for having fun!

Happy benchmarking!

P.S. First time I heard about benchmark_suite was at Toster conference in Moscow during the presentation by Jon Leighton, maintainer of ActiveRecord. Slides

Similar posts

Comments

makaroni4 commented about 1 year ago

E302c3320cd14b02cbe237b479d7f884?size=52

And one more example of testing classes:

require 'active_record'
require 'benchmark/ips'
require 'benchmark'

ActiveRecord::Base.establish_connection(
  adapter: "sqlite3",
  host: "localhost",
  database: "/Users/makaroni4/projects/gistflow/db/development.sqlite3",
  pool: 5,
  timeout: 5000
)

# if you don't have a table in your db, create it like this:
#
# ActiveRecord::Schema.define do 
#   create_table :posts do |t|
#     t.string :title 
#   end
# end

class Post < ActiveRecord::Base
end

p = Post.create(:title => 'title')

Benchmark.ips do |r|   
  r.report("[] ") do
    p[:title]
  end

  r.report("read_attribute ") do
    p.read_attribute :title
  end

  r.report("attribute ") do
    p.title
  end
end

# Calculating -------------------------------------
#                    []  10305 i/100ms
# read_attribute  10362 i/100ms
#         attribute  13395 i/100ms
# -------------------------------------------------
# []                      2607165 in   4.994431s
# read_attribute  2611224 in   4.997492s
#          attribute  6911820 in   4.991949s

releu commented about 1 year ago

757fb0d5ec7560b6f25f5bd98eadc020?size=52

Benchmarking always rule :)

eregon commented 5 months ago

0ea7f61aec8fee539be0cf39b7bab77c?size=52

Be aware the last example is mostly measuring yielding a block in a loop (the third form of #report can help a bit, but anyway addition is too fast and some implementations optimize it out as result is unused) and the one using N.times and benchmark-ips is inaccurate at least in error reporting (due to the so small number of iterations).

makaroni4 commented 5 months ago

E302c3320cd14b02cbe237b479d7f884?size=52

@eregon so how can we run this comparison properly?

eregon commented 5 months ago

0ea7f61aec8fee539be0cf39b7bab77c?size=52

It is really hard to measure Fixnum#+ and #* because any execution to measure it will account very significant external factors, such as loops (if done in ruby, i+=1 will be in the loop, which is likely taking about the same time as 42+42), integer boxing/unboxing, etc. And to avoid aggressive optimizations, a result must be produced (and ideally used). So measuring precisely such fast operations with ruby tools is not realistic, but it is also very unlikely the bottleneck.

The example with String#+ / String#<< is mostly fine (without the N.times when using benchmark-ips), but the GC might take an important part of it and there are side effects so they can not be compared easily quantitatively.

codesavvy commented 5 months ago

474288f408c5950862a4b346b60201a1?size=52

Nice post.. :D

codesavvy
eregon
killthekitten
makaroni4
releu

About

Join gistflow developers community

  • Start your developer blog in a minute.
  • Share code-related posts.
  • Follow users, subscribe for tags.
  • Use gist.github.com gists to write posts.

Tags