Monday, April 11, 2011

Logging to a database

So you're writing a Mac or iOS app and you're contemplating sending your log statements to a database. There are many benefits to doing this, especially when one considers the purpose of the log statements: reading them.

Example 1:
You're told of some problem your boss had with the application. When did it happen? "Yesterday around noonish." Now if you're using log files then you find yourself opening a few of them to find the right one. After that you start scrolling to find where you should start reading. But with a database you can just query it, and tell it you want to see the log statements between 11 AM and 1 PM from yesterday. (An easy thing to do with the plethora of database tools at your disposal.) In addition to this you may be able to easily filter out log statements that don't have anything to do with the problem.
Example 2:
Your application crashed, and you want to send an automated crash report containing the log statements leading up to the crash. This is somewhat similar to the task above, except now you have to get the log statements programmatically. With log files this might be a bit of work. You'll have to "tail" the most recent log file. But how many lines do you want? It's a difficult question to answer. The best answer is likely, given the time from the crash report, all log statements several seconds before the crash up to the time of the crash. If you're doing this from log files, you now find yourself reading the file backwards and parsing timestamps. And don't forget to take into account the possibility that the log file was rolled during this time, so you might need to consult multiple log files. With a database, it's easy. Just send it a query.


Historically there have not been many logging frameworks for Mac/iOS developers. So most simply use NSLog, or design some primitive macros around it. But as your application or development shop matures, the need for more advanced logging arises. And this is where the trouble begins. Because sending your log statements to a file or database doesn't seem that difficult. "I'll just tweak my macros to also invoke..." And before you know it... BAM! You've just committed yourself to writing a fully-fledged logging framework. Because the devil is in the details, and it turns out that sending those log statements to a database ended up being a bit more difficult than you originally thought.

For starters, your log statements need to go to NSLog and the database. Because you still want your log statements to show up in the Xcode console when you're debugging. In addition to this, you don't want your database to grow infinitely large. At some point you should probably delete old log statements. The age cutoff is up to you, but in terms of implementation this means that you need to delete entries whose timestamp is older than the cutoff. How often do you run this cleanup? And if you're thinking about saving to the database for every log statement, think again! Because a save operation doesn't return until the data is safely on the disk. In other words, it's synchronous disk IO, it's slow, and it will significantly degrade your application's performance. So you should buffer your save operations. One possibility is to save every X log statements. But what if there's not a lot of logging. Another option is to save every Y seconds. But what if there's a ton of logging? A better solution would use a combination of the two, with configurable settings. And don't forget to flush when the application quits...

Luckily you don't have to worry about any of this! Because the Lumberjack Logging Framework will do all of it for you. It's a powerful logging framework, and it's an order of magnitude faster than NSLog. It supports sending your log statements to just about anywhere, and it comes with sample projects showing how to log to a CoreData database or a raw SQLite database.

For more information about Lumberjack see the Getting Started page, or see all the wiki articles.

The database logging examples can be found in the folders:
- Xcode/CoreDataLogger
- Xcode/SQLiteLogger

 

No comments: