Reducing memory usage of rsyslog

Posted by on

I’ve never been a big fan of rsyslog, preferring syslog-ng instead. However, I am giving rsyslog another chance. One of my biggest problems with rsyslog has been it’s high memory usage, documented by many people in many complaints that can be found using google. Now this emmory is usually, though not always, virtual and not resident. That means rsyslog is not actually hogging memory but merely causes reporting problems when searching for processes with high memory usage as it will often show up. However, I’m trying a new solution that I hope will improve the situation. From the rsyslog wiki:

rsylsog is a (potentially massively) multi-threaded syslogd. Each of the threads requires a runtime stack. Rsyslog uses no specific stack allocation and sticks with the OS default. Seen in practice have stack allocations of 8 to 10 MB per thread. In a process trace, this can look like a memory leak.

—Reducing memory usage

So how big is the default stack for rsyslog:

cat /proc/`pidof rsyslogd`/limits

On all of the 64-bit Ubuntu systems I’ve tested the answer is:

Max stack size 8388608 unlimited bytes

So 8MB. The easiest fix I discovered is this:

Edit /etc/default/rsyslog and add the line:

ulimit -S -s 128

This sets the soft limit for stack size to 128K. Then restart:

restart rsyslog

Then verify:

cat /proc/`pidof rsyslogd`/limits

Max stack size 131072 unlimited bytes

Stumbling on cool things

Posted by on

I’m interested in emulators, always have been, and I was wondering if anyone had used LLVM to write an emulator. I came across this highly detailed account which was a fun read:

Statically Recompiling NES Games into Native Executables with LLVM and Go

It concludes stating that static recompilation is probably less practical than JIT, which I think is the correct conclusion. Still, nice to have someone go through the effort of building something just to show how it can be done.

New Keyboard

Posted by on

I’m looking at keyboards as my BTC 6100C is starting to develop some key issues. I’ve been pretty happy with the 6100C but it’s not longer made and BTC is not making any wired compact keyboards at this time. Here’s what I’ve found:

  • A4Tech KL-5 and variants ($40) – these are nearly identical to the BTC 6100C so probably have the same manufacturer. However the A4Tech removes the volume control buttons and the other hotkeys are not particularly useful without remapping. I’m considering it a fallback option is I don’t like the feel of other options.
  • Genius LuxeMate i200 ($19) – I like the layout and I like the fact that it actually has a zoomable picture on amazon to look at the layout details closely.
  • GearHead – 89-Key Mini USB Windows® Keyboard ($15 plus shipping) – Looks like a very similar layout to what I have but nothing in the description sells it over other optins.
  • SIIG JK-US0312-S1 ($22) – Better hotkeys and similar layout to the BTC 6100C.
  • GMYLE® Ultra Thin Wired USB Mini Keyboard – Only 78 keys, thought Fn + directions for Page Up/Page Down/Home/End makes a lot of sense.
  • Boxcave 78 Key Wired USB Mini Slim Keyboard ($20) – identical to GMYLE
  • Perixx PERIBOARD-407B
  • https://stackoverflow.com/questions/414703/compact-keyboard-recommendations

Important detail for Rails apps at a sub-uri

Posted by on

When using passenger with an app deployed to a sub-uri most details of app paths are handled automatically. However, when deploying an app the precompiled assets using helpers such as asset_path will get the wrong path because it doesn’t know about the sub-uri. The solution is to set Rails.application.config.action_controller.relative_url_root in an initializer or environment to the sub-uri path. I learned this from the font-awesome rails documentation but this is a common issue as I found a lot of people running into similar problems wherever assets where being precompiled.

Be clear this is:

Rails.application.config.action_controller.relative_url_root

and not:

Rails.application.config.relative_url_root

Also this can be accomplished by setting the environmental variable RAILS_RELATIVE_URL_ROOT but I think that will usually be more work but maybe there are instances where that is the better solution.

It would be wise to include this in the passenger documentation for deploying to a sub-uri as this is where most people could be made aware of the importance of this detail and how not addressing it could lead to problems later.

Ubuntu Boot_Degraded

Posted by on

As far as I’m concerned the decision by the Ubuntu Team to default to failing to boot when a raid array is degrade is a complete mistake. When your decision is the opposite of what everyone else has chosen to do in a given situation, you’re probably doing it wrong. Having an option for people to choose to fail to boot on a degraded array is a great idea and one that would probably almost never be used. But I’m all for options, and also for sensible defaults. So on any Ubuntu system where you’re running mdadm raid don’t forget to do either:

  • sudo dpkg-reconfigure mdadm
  • OR
  • update /etc/initramfs-tools/conf.d/mdadm and set BOOT_DEGRADED=true

Followed by:

  • sudo update-initramfs -u -k all

Every Ubuntu system with mdadm RAID, every time.

Ruby best practices: count, length, size

Posted by on

When to use which:

Hash

  • size and length both call RHASH_SIZE which is O(1)
  • count is enumerable.count which is O(n) – use only with block
  • best practice: length without block, count with block

Array

  • size is an alias for length
  • length calls RARRAY_LEN which is O(1)
  • count is O(n) – use only with block
  • best practice: length without block, count with block

ActiveRecord

  • count and size create a COUNT query which is often faster than alternative
  • length creates an array and calls length which is usually slower
  • best practice: count

Performance problems with FreeTDS

Posted by on

So FreeTDS is the glue between a linux systems and SQL Server, among other things. For my purposes I’m mainly using it to run Rails with a SQL Server backend. My understanding is that the alternatives to FreeTDS are Microsoft’s SQL Server ODBC Driver 1.0 for Linux or, if using JRuby, Microsoft’s JDBC Driver for SQL Server. But why am I even looking at alternatives? Well, FreeTDS has a performance problem when inserting large numbers of records. This bug may not be universal, which is to say it might only appear in certain contexts, but it is significant. Basically when bulk inserting records performance is abysmally slow. I’ve seen this within Rails which made me think maybe the problem was in TinyTDS or ActiveRecord SQL Server Adapter. However, I’ve noticed the problem with FreeTDS binaries tsq, bsqldb, and fisql. That makes me think the problem is in FreeTDS, at least the stable 0.91 release which was released in August 2011 though has been patched subsequently. It’s possible that the current 0.95 version will resolve these problems but I have not yet tested it. There is one binary not affected by this problem and it is freebcp. However, I have been finding that freebcp has it’s own bugs/quirks/idiosyncrasies or perhaps is exposing those from the underlying freetds code. In any case, freebcp is not a great solution, but for bulk data transfer from linux to SQL Server it seems to be the only game in town.

Rails Database Config Parameters for PG Gem

Posted by on

I’ve noted how difficult it is to get a complete list of parameters that can be used in the Rails’ config.database.yml file. I understand this is because there are different parameters for the different library layers. Still, I would like a more complete list somewhere, especially given that the parameter names can be confusingly similar but different between adapters (eg: timeout, connection_timeout, login_timeout, connect_timeout) For PG the disparate parameters can be found here:

I may create similar list for SQL Server.