haruki zaemon

The more I use Rails (and the more plugins I create) the more quirks I find.

Imagine I have a one:many relationship between Country and State:

State.belongs_to :country
Country.has_many :states

We then issue the following sequence of statements (I’ve interleaved the output of tailing the development log):

c = Country.find_by_name('Australia')
**  Country Load (0.006506)   SELECT * FROM countries WHERE (countries."name" = 'Australia' ) LIMIT 1**
s = c.states.build(:name => 'Victoria', :abbreviation => 'VIC')s.country
**  Country Load (0.009738)   SELECT * FROM countries WHERE (countries.id = 1)**

Notice the SELECT to find the country? Now why would that be necessary? I just used .states.build on the country. I would have thought that would set the association but that doesn’t appear to be the case.

Looking at the code, my suspicions were confirmed: only the parent’s id is set. That seems decidedly odd given that we know for a fact the parent exists – we just used it to create the child.

So anyway, I’m pretty sure this is considered a “feature” but to be honest, I can’t see why it is desired behaviour over and above the fact that doing otherwise would be more work and why would you need this if you already have the parent yada, yada, yada.

Well, for a start, I’d like this behaviour because I’d like to use validates_presence_of on foreign-keys and have it work for newly constructed graphs. Usually this barfs no matter what but I concocted a work-around last night and committed it to my Foreign Key Associations plugin which, if done manually, would look something like this:

class State < ActiveRecord::Base
  validates_presence_of :country_id, :if => lambda { |record| record.country.nil? }
  ...
end

Essentially this says to validate the presence of country_id but only if there isn’t an associated country. This means that for cases where the parent record is also new, the validation checks for the presence of the associated object rather than the foreign-key column. If you had simply used validates_presence_of :country_id then save would fail because country_id was still nil.

OK that’s all very well and good but it still doesn’t help because, as shown above, the association isn’t set anyway. So, I’m now back to manually setting the association; at least the validation works hehe

I’m sure someone far smarter than I will point out why the behaviour as it stands is obviously the most appropriate and that no one in their right mind would want to do anything else, of course ;-)

Everyone else’s favourite database just gave me the shits, again!

As part of my Schema Validations plugin for rails, I needed to see if a column has a default value. If it does, then there’s no point in adding a validates_presence_of as the database will add one in. Ok, sounds sensible. Works just fine under PostgreSQL but my tests were failing when run against MySQL. Specifically, there was no validation being added for integer columns marked as NOT NULL. Huh?!

After a little investigation, I noticed that the meta-data that rails was collecting for mandiatory integer columns included a default of 0. So I looked in the test database and sure enough the columns all had a default of 0. But how? Why? I didn’t put a default in my migrations…

A little more investigation and I noticed that the schema dump that is generated out of the development database and then run against the test database did indeed include the very same defaults. I then looked in the development database and to my surprise found no such defaults there. Aha! Mystery solved I presumed. Rails must have a bug for MySQL.

So I go and look at the code but alas, the code is the same for both PostgreSQL and MySQL. Something else must be happening. Time to get down and dirty on the command-line.

mysql> create table foo (col1 int, col2 int not null, col3 int default null) engine=InnoDB;mysql> show columns from foo;+ -- -- -- -+ -- -- -- ---+ -- -- -- +- -- --+ -- -- -- ---+ -- -- -- -+| Field | Type    | Null | Key | Default | Extra |+ -- -- -- -+ -- -- -- ---+ -- -- -- +- -- --+ -- -- -- ---+ -- -- -- -+| col1  | int(11) | YES  |     | NULL    |       || col2  | int(11) | NO   |     |         |       || col3  | int(11) | YES  |     | NULL    |       |+ -- -- -- -+ -- -- -- ---+ -- -- -- +- -- --+ -- -- -- ---+ -- -- -- -+

If I have a nullable column then the default default (if that makes sense) is NULL. If I mark a column as mandiatory, the default default is…an empty string!? I wonder what would happen if I tried inserting a row and letting MySQL default all the values:

mysql> insert into foo () values ();mysql> select * from foo;+ -- -- -- +- -- ---+ -- -- -- +| col1 | col2 | col3 |+ -- -- -- +- -- ---+ -- -- -- +| NULL |    0 | NULL |+ -- -- -- +- -- ---+ -- -- -- +

You have to be shitting me! I attempted to insert a row into a table without specifying a value for a column that is marked as NOT NULL and it inserts 0!? Hold on a second…what if I force the default to be NULL so that it behaves just like every other sensible database on the planet:

mysql> create table bar (col1 int not null default null) engine=InnoDB;ERROR 1067 (42000): Invalid default value for 'col1'

Egads! OK let me try that in PostgreSQL:

psql=# create table bar (col1 int not null default null);CREATE TABLE

Thank-you!

Sure, I could make the assumption that 0 was never going to be a valid identifier for a record in another table but why should I have to? As far as I can tell, MySQL is just making shit up! No wonder my brother says it reminds him of using Microsoft Access.

So, now I’m left with the task of working out how to patch rails to get around this. I think I’ll just have to presume that empty strings are equivalant to NULL for manditory columns. Sheesh.

I mentioned in my previous entry that I’d done quite a bit of refactoring of the plugins. Among the various changes that will affect developers using them are:

• Schema Defining (schema_defining) has been deleted;
• Foreign Key Support (foreign_key_support) has been deleted; and
• RedHill on Rails Core (redhillonrails_core) has been added to replace the previous two as well as subsuming some of the more generic functionality from other plugins.

So, why all these changes?

The main reason is manageability. We’re actually eating our own dog food and using these plugins in production applications and we’re adding functionality at quite a surprising rate. Each time we add something, we first put it into the plugin that needs it directly. That works great for a while but then, someday, we decide we need that functionality in two or more plugins. What to do?

Our original idea had been to create new plugins and this worked for us up to a point. Unfortunately, of late, the number of extra plugins – with very specific functionality mind you – was just getting out of hand and needed to be simplified.

In the end, we decided on a two-tiered approach to plugins: those which add functionality but no (or at least minimal) behaviour; and those that add behavioural magic.

As an example, the new core plugin adds functionality to manage foreign keys, lookup indexes, add unique column meta-data, etc. but doesn’t do anything particularly magic that will affect the running of your application.

On the other hand, the foreign key migrations, foreign key associations, schema validations, etc. plugins – which all rely on core – add funky rails magic to automatically generate foreign keys, associations, model validation, etc.

Another change we made was in the way documentation is generated. We used to manually generate a nice HTML file containing all the plugins. This was becoming rather tedious and meant that the documentation was often quite out of date. We’ve now remedied this with a nice ruby script using Erb and RDoc to generate the online documentation directly from the README files.

I also mentioned previously that we’ve added “lots” of tests. I say lots because we’re still playing catchup so relatively, there are lots but we still need lots more. As a group of developers that are ardent TDD evangelists, the conspicuous lack of tests was somewhat embarrassing to say the least. Unfortunately, testing plugins (especially those related to schema and database) is pretty difficult so we opted to bypass the whole problem and just create a standard rails app with standard rails tests and all is well again.

And lastly, besides all the extrat features we’ve added (see the CHANGELOGs for the specific plugins), you’ll notice that the subversion URL has changed slightly – it used to contain an extra slash (/) which was not only unnecessary but caused SVN to regularly crap out.

My aplogies to all those that have been trying to keep up but we hope that’s the last of it. From now on, we’ll continue to beef up core as we need and then add plugins only when we need new behaviour.

Of course we’ll always reserve the right to change our minds ;-)

I’ve done quite a bit of refactoring of my Ruby on Rails plugins lately which, unfortunately, broke some stuff (thanks to all those that let me know) but the upshot is a much cleaner division of responsibility between plugins; and some sorely needed unit tests.

Another of the benefits from all of this was yet another plugin, this time to automatically generate associations based on foreign-keys.

For example, given a foreign-key from a customer_id column in an orders table to an id column in a customers table, the plugin generates:

• Order.belongs_to :customer; and
• Customer.has_many :orders.

(In the near future we intend to support has_one associations for foreign-key columns having a unique index.).

If there is a uniqueness constraint – eg unique index – on a foreign-key column, then the plugin will generate a has_one instead of a has_many.

For example, given a foreign-key from an order_id column with a uniqueness constraint in an invoices table to an id column in an orders table, the plugin generates:

• Invoice.belongs_to :order; and
• Order.has_one :invoice.

You can download the latest version directly from svn://rubyforge.org//var/svn/redhillonrails/trunk/vendor/plugins/foreign_key_associations

For all those that have asked for pure HTTP access, I hear you and I’m working on it. (It seems ./script/plugin install doesn’t understand the format of the browse repository pages on RubyForge. DOH!)

I wrote a while ago on utilising transactional DDL in your ruby on rails migration scripts so I decided to create a plugin.

In a nutshell:

Transactional Migrations is a plugin that ensures your migration scripts – both up and down – run within a transaction. When used in conjunction with a database that supports transactional Data Definition Language (DDL) – such as PostgreSQL – this ensures that if any statement within your migration script fails, the entire script is rolled-back.

The National Breast Cancer Foundation is Australia’s national fundraising body for breast cancer research and is hosting a charity event at beautiful Rippon Lea Estate in Melbourne on the 8th September, 2006 to raise money.

All the details are available online so get out your $100, put on some party wear and come and support a wonderful charity.

And yes, even geeks are allowed!

Yet another plea: Please don’t add foreign-key migrations, schema validations or for that matter acts_as_taggable or any significant number of the myriad plugins that are now available. Leave RoR as lean and as mean as possible. By all means change your assumptions and your opinions but don’t allow Rails to become the Micro$oft Word of the Ruby world – bloated and with features that < 1% of the community ever use.

I’m pretty opinionated. DHH is obviously pretty opinionated. That doesn’t mean I necessarily agree with his opinions – I clearly think foreign-keys are important – but that doesn’t prevent me from using RoR. In fact, quite the contrary. Precisely because DHH is so reticent to adding every new feature under the sun into RoR, the current feature set appeals to most of the developers who use it. This is not to say that Rails is in anyway fully-featured but what is there, most people use. So what about all that neat stuff that we all think is great and absolutely necessary but with which DHH and, no doubt, a non-trivial number of other developers in the community disagrees?

I think my favourite feature of RoR is the very sophisticated plugin model. With a little thought and imagination, it’s pretty easy to implement just about any extension imaginable. And this is where the power lies in being lean, mean and opinionated. It’s much easier to add features than to take them away – actually it’s pretty easy to take them away too but it can get pretty ugly and besides, who wants to spend their time writing plugins to disable functionality? In fact I like plugins so much, I’ve started thinking about my applications as collections of plugins. Plugins work, there are lots of them, they allow you to add features that no one ever dreamed of and then, with very little effort you can, if you’re a good sort, give something back to the development community.

You probably don’t want to use much (if any) of the stuff that I think is useful and I sure as hell don’t want your manky ideas cluttering up what continues to be my favourite development environment. So, please stop inundating the RoR Trac with every little thing that you believe to be 100% necessary and start building and publishing plugins.

I’m not sure anyone else really bothers but today I decided I’d try listing to the Text-to-Speech version of this blog by clicking on the Listen to this article link. Apparently the software they use hasn’t caught up with Plugin as a commonly used variation of the hyphenated Plug-in. I was somewhat amused therefore, to hear about the Ruby on Rails ploojins :).

After listening to Prag Dave’s Keynote Speech this afternoon, I was motivated to implement some of the things he’d been asking for. Here’s my first cut at it.

As the doco says, the plugin reads some – ok only one at the moment but we’ll see how many others I get done before the beer runs out – database column constraints and tries to apply the closest corresponding rails validation. The first one I implemented reads the NOT NULL constraints against columns and generates a corresponding validates_presence_of.

I literally just whipped it up with no tests or what-not and I’ve only played with it against PostgreSQL, so if it has bugs or behaves oddly for whatever reason, please let me know, send me as much info as possible and I’ll make it work. Nothing better than having real people testing it for me ;-)

UPDATE: OK, so far the beer has lasted long enough to implement validation of numbers (including specific support for integers) and lengths of strings.

UPDATE: Now calls validates_presence_of anytime you declare a belongs_to association for a NOT NULL foreign-key column.

UPDATE: Single-column unique indexes are now converted to validates_uniqueness_of.

This morning, my brother was sitting at his laptop attempting to get FreeBSD to mount a ReiserFS partition without much success – some permissions problem that means he can mount it as root but not from fstab – when his girlfriend sat down beside him, peered over, and asked “What’s that weird devil number?” Naturally we both had a little WTF moment. “See”, she continued, “it’s even called devil!”

On closer inspection, it seems the file permissions for /dev are, understandably, 666. Riiiiiigghhht!

Be warned, this is somewhat of a stream-of-consciousness even if a little edited.

I was sitting at work today pondering, among other things, some particularly awful code that we happened upon – so what’s new? The code in question relied upon multiple columns in a table to hold configuration information when really what was needed were multiple rows – something like a properties file but in a database. I then started to think that I pretty much always start out wanting my data to be “properly” normalised without regard to performance considerations. As Knuth is oft-quoted as saying, “premature optimisation is the root of all evil.”

One thing I really like is when a development environment allows me to easily layer on performance enhancing behaviour such as caching, without unduly interfering in the core design of the system. In a Rails for example, I can build a completely dynamic application and strategically add caching where necessary for content that is, for the most part, static. If the data on which that content relies changes, expire the cache and it’ll be re-generated.

At the same time, I was pondering why it is that I like Ruby so much and why I believe it really makes development environments such as Rails possible; in a way that Java just doesn’t. Although a topic for another blog entry, one of the reasons I considered was it’s dynamic (ie non-compiled) nature. That I can change a piece of code and immediately run it without a compilation step is, IMHO, a big factor. (I’ll grant you that Eclipse goes a long way towards achieving this for Java with its continuous compilation but I feel there is more to it than just that.)

It then occurred to me that compilation, or, more specifically, the translation of source-code into some executable form and stored in some more persistent manner (such as files on a hard disk) is really a form of caching.

To see what I mean, imagine you started with a fully, 100% interpreted language where each statement is re-parsed each time it is executed. Ie. no caching. To this you then decide to convert the source-code into an intermediate, in-memory, form to remove the need to re-parse the source-code each time. This is a first-order cache. Finally, you decide to write this intermediate form to a more persistent medium so as to remove entirely the need for conversion from raw source-code to an executable form. This is a second-order caching. You could of course further translate the intermediate form into native machine code or any number of intermediate formats. And, of course, once in memory, the intermediate format could be dynamically translated into machine code – such as a Just-In-Time compiler or Hot-Spot compiler does.

The point I’m trying to make however is not how efficient one can make the execution process but simply that it is entirely possible to execute the raw source-code with a very simple parser and interpreter. The fact that we choose to further translate that source-code to forms specifically designed to improve execution efficiency is an optimisation and, in many ways, not so different to storing the output from a Rails action on the server, or a web browser caching an image or an HTML page.

Where the analogy breaks down of course, is that compilers often provide other benefits such as detecting poor or incorrect use of the langauge. That said, some of these benefits are lost – or for that matter not even useful – when using languages such as Ruby and yet other benefits might be found in static analysis tools.

So, what does this all have to do with anything?

That’s a very good question to which the answer may well be “not very much at all” however, I did find it interesting to ponder, as one does ;-)

One thing that I have noticed though, speaking of source-code, is there exists a distinct difference in the way the Ruby versus Java communities treat the sharing of code.

In the Java community, sharing of code is largely done by means of pre-compiled libraries with accompanying documentation. Sure, we have open source projects which make the source-code freely available for (ab)use. But, and I admit to having been guilty of this myself, the general feeling is don’t modify the source unless you really need to; treat the binary distribution as a black box. And perhaps for good reason?

In the Ruby (and apparently Smalltalk) community, the philosophy is more one of source-code sharing. Need a Rails plugin? download the source-code into your source-tree; Looking for some missing Ruby functionality? Native extensions aside, what you’ll get is a bunch of source-code. In most cases the source-code also comes annotated with RDoc – the Ruby equivalent of Javadoc– but even still, being confident to look into the source-code is, IMHO, a bonus.

Of course, there are those who would argue that sharing of raw source-code isn’t always a commercially viable option and perhaps they’re right.

In any event, I’m not sure I have enough experience in either yet to have formed an opinion as to which (or even if) one approach is better than the other but I do wonder if the fact that one is primarily treated as a compiled language whereas the other is essentially treated as interpreted leads to at least some of the differences in culture that I see or if I have cause and effect around the wrong way.

I’m back doing some Java work at the moment and although coding in Java feels somewhat like trying to speak with a mouthful of peanuts – compared with Ruby – I’m rather enjoying it for some perverse reason. That said, my long-time trusty steed in such endevours is really beginning to annoy me. IntelliJ, once the lean, mean, IDE has porked out considerably over the last few releases. It seems that in competing with Eclipse, it is becoming Eclipse: fat, bloated and slow.

IntelliJ has become a bit like Micro$oft Word: I only ever use 5% of the features – and that’s just the features of which I’m actually aware; I’m beginning to need the 2GB of RAM in my laptop just for IntelliJ; oh, and SVN+SSH integration just plain sucks. I’ll be sticking with the command-line version thank you very much.

It’s all very well and good to have all these nifty tools, etc. in the IDE but seriously, I never use them. Nor, for that matter, do I want to get into the habit of doing so for one simple reason: I have command-line build tools so that I may run the build on any machine, at any time of the day, even gasp when there is no IDE involved nor a programmer to be found.

So here’s a plea to JetBrains: I paid good money for your software and I’ll be more than happy to continue paying good money if you’ll please oh please let me turn off all those features that make me feel more like I’m using WSAD.

What’s next, Perspectives? Gadzeeks!

I’m usually reticent about making politically motivated blog entries but today I was having a to-and-fro via email with a friend of mine when she made a statement that, especially in light of the hoopla surrounding a certain TV show here in Australia, I felt was too good not to quote:

It’s a pity that the people who watch Big Brother don’t equally scrutinise the behaviour exhibited on that other reality TV show, ‘Parliament Question Time’.

Here, here.

When I moved to FeedBurner I hardly ever used delicious and I thought it might be interesting to have any new links included as part of my feed. These days however, I find myself adding links to delicious on a regular basis and my feed has been somewhat flooded as a consequence. So, by request, I’ve removed the links from the feed. You can always subscribe to them using delicious anyway if you really cared (not that you probably did).

Ever since I made Simian available, I’ve had a strong belief that the software should be available for download in a non-crippled form. That is, whether you pay for it or not, makes no difference to the functionality; you are however left with a clean conscience :)

Unfortunately, this is something that many customers find quite difficult to come to terms with. Many of them, after parting with a relatively small sum, insist they be sent a license file, software keys, or something else tangible for their money. I’m at the point now where I’m considering simply sending out a file containing 128-bytes of randomly generated gibberish that they can place in the software directory just to give them that warm, fuzzy feeling.

Did I miss some unwritten rule that says a polite email thanking them for their purchase isn’t enough?

This relatively unknown feature of subversion allows you to include one subversion repository URL inside another. By that I don’t mean export and add, rather svn:externals allows you to create a virtual directory that links to an external repository.

I’m sure there are plenty of other uses but I mostly use svn:externals for adding plugins to my rails projects. For example, if you wanted to link to the Foreign Key Migrations plugin, from inside your rails project directory you would type:

simon$ svn propedit svn:externals vendor/plugins/

Then add the following lines:

foreign_key_migrations svn://rubyforge.org//var/svn/redhillonrails/trunk/vendor/plugins/foreign_key_migrationsschema_defining svn://rubyforge.org//var/svn/redhillonrails/trunk/vendor/plugins/schema_defining

And voila! Now when you update your project you’ll see something like:

simon$ svn upFetching external item into 'vendor/plugins/foreign_key_migrations'External at revision 25.Fetching external item into 'vendor/plugins/schema_defining'External at revision 25.At revision 2533.simon$

Notice how subversion updates the virtual directories as well.

Be aware though that this does mean you’ll always get the latest code when you update. This is exactly what I want but may not suitable on all projects. Sometimes a simple svn export is more appropriate.

We’re currently toying with the possibility of using svn:externals to share common code between some projects that are presently all in one big code-soup project but will be split up over the next couple of months. I’ll let you know what we decide to do in the end.

Besides the unfortunate use of a z where there really ought to be an s (yes, yes, I’m Australian), what does this word really mean?

pluralize v. tr.* To make plural.

• Grammar. To express in the plural.

OK, that’s pretty much what I thought it meant. So then riddle me this Batman:

simon$ ./script/consoleLoading development environment.>> "country".pluralize=> "countries">>

Good good. Just as expected. So, now let’s try something else:

>> "countries".pluralize=> **"country"**>>

Huh??? Since when did country become the plural of countries?

Ok so I’m an Aikido guy but I’m sure this will amuse most non-aikido people as well. Apparently it’s legit! So now I can’t help but make up funny movie titles:

Pushed to the brink by the death of his movie career, a down and out martial artist gives it one last go…Steven Seagal in: Out of Tune…coming to a cinema near you.

Just a quick update on my plugins. Two of them – Foreign Key Migrations and Row Version Migrations – needed to know if they were being run as part of ActiveRecord::Schema.define() so I extracted the common code into yet another plugin: Schema Defining. This new plugin provides a class method – ActiveRecord::Schema.defining? – that returns true if ActiveRecord::Schema.define() is currently running; false otherwise. Probably not a hugely useful plugin in the main but certainly a must have if, like me, you have a number of migration plugins that need to alter their behaviour accordingly.

The more I use Rails, the more plugins I seem to create. Each time I start a new project, almost the first thing I find I want to do is to copy a chunk of code from an existing project. The moment that happens, I think: Plugin!

So here it is. A plugin that automatically generates the following row version columns for every table:

:created_at,   :datetime,  :null => false

Not so difficult. In fact it seems almost trivial (as usual) but it means all my tables now get date/time stamps and optimistic locking for free.

Enjoy. As always, feedback, suggestions, patches, criticism, all welcome.