How to rotate Rails logs

How to rotate Rails logs

Piping colors with xargs in the unix terminal

Unix terminals are cool, no doubt: you can even pipe commands. But sometimes something goes wrong, and your’re prompted with strange errors.

Last week I needed to free some space on a few remote servers so I started poking with du. I found out the information I needed by piping ls with du with the command ls | xargs du -sh. When I got to the second server, I started to get crazy errors:

$ ls | xargs du -sh
du: cannot access `33[0m33[01;34mapp_link33[0m': No such file or directory
du: cannot access `33[01;34mbackups33[0m': No such file or directory
du: cannot access `33[01;34mbin33[0m': No such file or directory

The directory structure was pretty simple here and I initally saw no reason for the errors:

$ ls 
app_link  backups   bin

What you cannot see here in my post is color… in fact the terminal output on that server was in glorious rainbow colors. Those crazy codes that showed on the du errors were actually the shell representation of those colors. I could switch them off (maybe unaliasing ls?) but they looked good and humans generally prefer colored text rather than dull black characters, especially in the terminal, so I removed them by adding in the pipe chain a clever sed command:

$ ls | sed -r "s/x1B[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" | xargs du -sh
4,00K all_link
7,9M  backups
3,2M  bin

and that fixed the issue.

Fix

Fix

What is a Gemfile

What is a Gemfile

Don’t use before_action to load data

Don’t use before_action to load data

My misadventures while migrating a rails database from Sqlite3 to Mysql

Last week I needed to migrate a database from sqlite to mysql. This is a legacy rails project that uses 2 databases. One of them was completely managed by our customer (data, schema and server machine) on the Oracle platform. Recently we migrated this database from Oracle to sqlite, so at the moment the project is using two sqlite3 databases. We now needed to use mysql.

No schema was provided for the second db. My first option was to use the taps gem for the migration, I had used this gem on heroku in the past to import remote databases to my local machine and vice versa, but while looking at it I discovered that the migration could be even easier using the sequel gem, which by the way taps relies on. So I created the empty mysql db and entered this command on the terminal:

sequel -C sqlite://db/oracle_dump.sqlite3 mysql://root:@localhost/oracle_dump

Unfortunately, the process failed.

I spent some time trying to figure out what went wrong with no luck. Eventually, while inspecting the db schema, I found out there were a few primary key column names that were empty strings (yes! “” was the name of the column) and this was for sure an issue when trying to replicate the schema on mysql.

After renaming the empty columns to “id” I was able to fully recreate the schema on mysql with the command above, but the following step (data import) failed miserably: only a single record was created. The error reported was:

Error: Sequel::UniqueConstraintViolation: Mysql::Error: Duplicate entry '1' for key 'PRIMARY'

How could this be possible? All record ids in the source table were unique, I knew this for sure, it was fine with sqlite and oracle! But wait, the first record had a id value of 0! That was the reason for the failure: when mysql received the record data with id zero, it changed the id to “1” because by default zero is not allowed as primary key value. Later, when it was the turn of the actual record with id “1” to be inserted, mysql complained that the “1” key number was already taken. The solution was to temporarily update a mysql global variable to allow the insertion of zero values for primary keys: the variable named sql_mode.

I checked the current value of the variable:


mysql> SHOW GLOBAL VARIABLES WHERE Variable_name = "sql_mode";
+---------------+--------------------------------------------+
| Variable_name | Value                                      |
+---------------+--------------------------------------------+
| sql_mode      | STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION |
+---------------+--------------------------------------------+
1 row in set (0,00 sec)

And added the NO_AUTO_VALUE_ON_ZERO value to it:

SET GLOBAL SQL_MODE = "NO_AUTO_VALUE_ON_ZERO,STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION";

After this final fix the sequel import process worked perfectly. I later returned the global variable to its original value:

SET GLOBAL SQL_MODE = "STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION";

So, be warned: empty column names and zero value primary keys can make your database data transfer a bit though 😉

Picking a gem for fun and learning

Recently I’ve been working on a small rails project that handles uploads and imports of Microsoft Excel files.
File upload is a very common scenario and, as most rails developers already know, there are plenty of gems to ease the job. Some of them like Paperclip, Carrierwave or Dragonfly are so well established that you really can’t go wrong with any of them, they’re complete and easy to use, with great documentation and a huge user base; in short, they’re a perfect tool for the job.

Parsing MS Excel files is not as common but I have already done it a few times through the years and there are many gems for this task as well, new and old. The good old spreadsheet gem is the one I used most in the past, but the most promising and interesting one is axlsx, to be paired with axlsx_rails if you plan to use it inside a rails application.

I eventually decided to use a different one, a small unpolished gem named creek. Why? One reason is that, unlike any of its competitors, it builds an hash structure for each document row instead of using an array, so you get something like this:


[
  {"A1" => "First Row, first col", "B1" => "First row, second col", "C1" => nil}, 
  {"A2" => "Second Row, first col", "B2" => "Second Row, second col", "C2" => nil}
]

instead of:
[
  ["First Row, first col", "First row, second col", nil],
  ["Second Row, first col", "Second Row, second col", nil]
]

I think this is a nice feature for speeding up the debugging process because having explicit cell coordinates makes the mapping to the original document so much easier, at least for me, especially when the XLS file has many columns and data is repeated or very similar among them.

Anyway, there are many other reasons I can think of, for example:

  • the gem is two years old, which means that it’s not in its early development stage so I can assume that most of the relevant bugs have already been worked out
  • github shows recent activities for this gem, which means that it’s not abadonware and the author is still taking care of the maintenance, answering promptly to programmers issues and merge requests
  • the codebase is simple and small, so modifications and updates should be easy when needed
  • there are a few tests written with rspec, which means that the gem developer follows the ruby community best practices and I can easily add new ones following the existing patterns
  • it’s easier to get involved when the project is very small: you can easily think about new features to add or make some simple refactoring activities which will probably be welcomed by the author

I think the last point is one of the nicest facets of open source programming: it’s easy to get involved and contribute with some small modifications that can improve the original codebase, make new acquantaintances in the process, and start a learning experience for some of the involved programmers, may that person be you or the original developer. If you’re more skilled you probably can teach some nifty pattern, trick or refactoring to the gem creator, while if he’s more skilled than you he can guide you in the process of improving the gem or show you why your modifications cannot be accepted (i.e. did you write tests for them? Are your changes too much coupled to your specific needs?).

Of course your mileage may vary, but I think that sometimes choosing a rather unknown gem instead of the most famous and celebrated one can be a great choice, especially when you’re on a small project and you feel confident that you can fix issues when they arise. It will surely be a more interesting and rewarding experience; of course we’re in the business because we are fast in delivering code that works, but nobody mandated that we should not have fun while doing it 😉

Happy coding!

Fun with Rails has_one through association

In my Rails developer career I had never felt the need to use the has_one through: association. Lately while working on a cool project I found out how useful and convenient this kind of association can be.

Let’s consider a simple use case. We have three ActiveRecord models: Player, Team and the join model Membership. A team has many players and a player can have many teams through its memberships. Of course a team has a captain, but this is something I will show later. Let’s start from the basic associations and the migration to create the tables:


class Player < ActiveRecord::Base
  has_many :memberships
  has_many :teams, through: :memberships
end

class Team < ActiveRecord::Base
  has_many :memberships
  has_many :players, through: :memberships
end

class Membership < ActiveRecord::Base
  belongs_to :team
  belongs_to :player
end

def change
  create_table :players do |t|
    t.string :name
  end

  create_table :teams do |t|
    t.string :name
  end

  create_table :memberships do |t|
    t.references :player, index: true
    t.references :team, index: true
    t.boolean :captain
  end
end

Alright, nothing really fancy here. When we want to associate a player to a particular team we don’t need to explicitly create the team membership record because rails can handle the underlying db details:


Membership.count
# => 0
player = Player.create name: 'Racco'
team = Team.create name: 'Mikamai'
team.players << player
Membership.count
# => 1

Let’s add another player to the team:


captain = Player.create name: 'Intini'
team.players << captain

That’s cool indeed. We how have two players in our team. You can tell by the variable name that we will soon make a captain of the last one that we created. Let’s complete the Team model in order to make the captain feature work:


class Team < ActiveRecord::Base
  has_many :memberships
  has_many :players, through: :memberships

  has_one :captain_membership, -> { where captain: true}, class_name: 'Membership'
  has_one :captain, through: :captain_membership, source: :player
end

Let’s put the new code to good use:


team.captain = captain

And that’s it! Rails will handle all the association details as before, but this time setting the membership captain boolean value to true as well.

Are we done? No, there’s a problem. Let’s see how many memberships we have now in the database:


 team.memberships.count
 # => 3

What about players?


team.players.count
# => 3

Ouch. We created only 2 players, so something went wrong.


team.players.map &:name
# => ["Racco", "Intini", "Intini"]

team.memberships.map { |membership| membership.player.name }
# => ["Racco", "Intini", "Intini"]

When we created the captain relation a new membership record was created as well, this time with the captain attribute set to true. One way to fix the players relationship is to pick players only once, making them unique, and this is quite easy to accomplish, we just need to update the team players relationship:


class Team < ActiveRecord::Base
  has_many :players, -> { uniq }, through: :memberships
end

Let’s try again with the new code:


team = Team.find_by_name 'mikamai'
team.players.count
# => 2

Success! Now, what happens if we change the captain? The existing captain membership record gets updated with a new player_id:


team.captain_membership 
# =>Membership id: 7, player_id: 6, team_id: 1, owner: true ...>

racco = Player.find_by_name 'racco'
team.captain = racco

team.captain_membership.reload
# =>Membership id: 7, player_id: 7, team_id: 1, owner: true ...> 

One last check then we’re done: what happens if the captain decides to leave the team? Will both relevant membership records be destroyed?


captain = team.captain
team.players.delete captain

team.players.count
# => 1
team.memberships.count
# => 1
team.captain 
# => nil

So everything works smoothly, no need to manually clear out stale association records. Did I mention that I love ActiveRecord?