Performance problems with FreeTDS

So FreeTDS is the glue between a linux systems and SQL Server, among other things. For my purposes I’m mainly using it to run Rails with a SQL Server backend. My understanding is that the alternatives to FreeTDS are Microsoft’s SQL Server ODBC Driver 1.0 for Linux or, if using JRuby, Microsoft’s JDBC Driver for SQL Server. But why am I even looking at alternatives? Well, FreeTDS has a performance problem when inserting large numbers of records. This bug may not be universal, which is to say it might only appear in certain contexts, but it is significant. Basically when bulk inserting records performance is abysmally slow. I’ve seen this within Rails which made me think maybe the problem was in TinyTDS or ActiveRecord SQL Server Adapter. However, I’ve noticed the problem with FreeTDS binaries tsq, bsqldb, and fisql. That makes me think the problem is in FreeTDS, at least the stable 0.91 release which was released in August 2011 though has been patched subsequently. It’s possible that the current 0.95 version will resolve these problems but I have not yet tested it. There is one binary not affected by this problem and it is freebcp. However, I have been finding that freebcp has it’s own bugs/quirks/idiosyncrasies or perhaps is exposing those from the underlying freetds code. In any case, freebcp is not a great solution, but for bulk data transfer from linux to SQL Server it seems to be the only game in town.

Rails Database Config Parameters for PG Gem

I’ve noted how difficult it is to get a complete list of parameters that can be used in the Rails’ config.database.yml file. I understand this is because there are different parameters for the different library layers. Still, I would like a more complete list somewhere, especially given that the parameter names can be confusingly similar but different between adapters (eg: timeout, connection_timeout, login_timeout, connect_timeout) For PG the disparate parameters can be found here:

I may create similar list for SQL Server.

Integrating R and Ruby

I did my time with rsruby, rinruby and Rserve-Ruby-client. In retrospect I should have trusted the Rserve-Ruby-client readme which details the problems with rsruby and rinruby.

  • rsruby – dealbreaker is that this is not stable, there are a few other downsides including complex data conversions and compilation issues but enough said.
  • rinruby – slow, but more importantly fails when assigning large data making it pretty much useless. See Bug #2 and Bug #13.

Solution: use Rserve and Rserve-Ruby-client. Both are actively maintained and stable.

Rserver can be installed on Debian/Ubuntu with:

apt-get install r-cran-rserve

Be aware that there is a bug in the Ubuntu package that requires fixing for rserve to work:

The easiest way to migrate a database from MySQL to PostgreSQL

I had to do this recently for a project. The difficult piece was sorting through the opinions of what the best tools and procedures for handling this migration was. So I tried a few until I found one that worked so well, so quickly that I decided it was worth sharing. Firs tthings I used that didn't work well and/or quickly and/or intuitively:

  • mysqldump with various options and various tweaks/scripts/editing of the dump file
  • pgloader

What did work well was py-mysql2pgsql. Once installed it was easy to set the configuration file options and run it. It worked without problem and I would use it again for this task. I can't comment as to whether it will handle all cases, this project didn't contain anything too fancy, but I would recommend it as a place to start:

Planning on buying a new monitor soon

I think my Dell 2005FPW is showing its age: I’m starting to get some rolling when the monitor is first powered on. It’s had a good run but I think I need to buy something or at least be prepared to buy before it becomes more problematic. I’ve resisted buying a new monitor because 1) if it’s not broken don’t fix it and 2) their has been little offered on the monitor market to encourage me to buy. There has been tremendous stagnation in the display world with regard to resolutions. I also strongly favor non-TN displays. The Dell 2005FPW uses PVA but IPS seems to have largely won out as the better display technology. Here’s the specs of my current monitor:

  • Dell 2005FPW – 20″ 1680×1050 PVA 16:10 (0.258 mm pitch)

What I have found is that in that size range there has been little improvement in resolution. I suppose this is understandable since 1080 is the number of lines considered to be HD and if you are watching HD content it is nice to do so without upsampling. So what I have found is the following size/resolution pairings:

New Flash Drives

I ordered the SanDisk Extreme USB 3.0 32GB. Half the price of the Lexar JumpDrive Triton and the best random write performance on a USB drive. Some research I did included:

The easiest way to get ruby 1.9 debugging functionality on Ubuntu 12.04

I’ve come across a number of posts giving instructions on how to get debugging to work with ruby 1.9.x and they all seem to involve downloading gems and installing them. Here is what I did on my Ubuntu 12.04 workstations that was very easy though not trivial to figure out.

sudo gem install linecache19 --pre
sudo gem install ruby-debug-base19 --pre -- --with-ruby-include=/usr/include/ruby-1.9.1/ruby-1.9.3-p0/
sudo gem install ruby-debug-base19x --pre
sudo gem install ruby-debug-ide --pre

Building xen-4.2.1 packages on Ubuntu 12.04

First install build dependencies. This list may not be complete as some necessary libraries and tools may have already been installed on my test machines. This is what I used to get the packages to build successfully:

  • apt-get install build-essential python-dev gettext bin86 bcc iasl uuid-dev libncurses5-dev pkg-config libglib2.0-dev libyajl-dev git gcc-multilib texinfo fakeroot

Then download, unpack, configure, and build:

  • wget
  • tar -zxf xen-4.2.1.tar.gz
  • cd xen-4.2.1
  • ./configure
  • make deb

Afterword you’ll be left with a shiny new package:

  • dist/xen-upstream-4.2.1.deb