Ruby & MySQL: How to handle missing elements while parsing XML file - mysql

Question

Currently I am trying to parse large xml file, Here is the how my xml file looks like:
<post>
<row Id="22" PostTypeId="2" ParentId="9" CreationDate="2008-08-01T12:07:19.500" Score="7" Body="<p>The best way that I know of because of leap years and everything is:</p>
<pre><code>DateTime birthDate = new DateTime(2000,3,1);<br>int age = (int)Math.Floor((DateTime.Now - birthDate).TotalDays / 365.25D);<br></code></pre>
<p>Hope this helps.</p>" OwnerUserId="17" LastEditorUserId="17" LastEditorDisplayName="Nick" LastEditDate="2008-08-01T15:26:37.087" LastActivityDate="2008-08-01T15:26:37.087" CommentCount="1" CommunityOwnedDate="2011-08-16T19:40:43.080" />
<row Id="29" PostTypeId="2" ParentId="13" CreationDate="2008-08-01T12:19:17.417" Score="18" Body="<p>There are no HTTP headers that will report the clients timezone so far although it has been suggested to include it in the HTTP specification.</p>
<p>If it was me, I would probably try to fetch the timezone using clientside JavaScript and then submit it to the server using Ajax or something.</p>" OwnerUserId="19" LastActivityDate="2008-08-01T12:19:17.417" CommentCount="0" />
</post>
Different between these two records in this XML file is that doesn't have LastEditDate element. I believe as a result of that I get the following error:
/ruby/1.9.2/ubuntuamd1/lib/ruby/1.9.1/date/format.rb:1031:in `dup': can't dup NilClass (TypeError)
from /soft/ruby/1.9.2/ubuntuamd1/lib/ruby/1.9.1/date/format.rb:1031:in `_parse'
from /soft/ruby/1.9.2/ubuntuamd1/lib/ruby/1.9.1/date.rb:1732:in `parse'
from load.rb:105:in `on_start_element'
from load.rb:165:in `parse'
Here is the code segment that its getting referred:
if element == 'row'
#post_st.execute(attributes['Id'], attributes['PostTypeId'], attributes['AcceptedAnswerId'], attributes['ParentId'], attributes['Score'], attributes['ViewCount'],
attributes['Body'], attributes['OwnerUserId'] == nil ? -1 : attributes['OwnerUserId'], attributes['LastEditorUserId'], attributes['LastEditorDisplayName'],
DateTime.parse(attributes['LastEditDate']).to_time.strftime("%F %T"), DateTime.parse(attributes['LastActivityDate']).to_time.strftime("%F %T"), attributes['Title'] == nil ? '' : attributes['Title'],
attributes['AnswerCount'] == nil ? 0 : attributes['AnswerCount'], attributes['CommentCount'] == nil ? 0 : attributes['CommentCount'],
attributes['FavoriteCount'] == nil ? 0 : attributes['FavoriteCount'], DateTime.parse(attributes['CreationDate']).to_time.strftime("%F %T"))
post_id = attributes['Id']
furthermore I think this is the line where I look for LastEditDate
DateTime.parse(attributes['LastEditDate']).to_time.strftime("%F %T"), DateTime.parse(attributes['LastActivityDate']).to_time.strftime("%F %T"), attributes['Title'] == nil ? '' : attributes['Title']
I guess since the element doesn't exist I get the above mentioned error. I was wondering how do I handle this scenario where if an element doesn't exist set it to a default value. Because while I am parsing these record I insert them into MySQL database. Which has following table structure:
+--------------------------+--------------+------+-----+---------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------------+--------------+------+-----+---------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | |
| post_type_id | int(11) | NO | | NULL | |
| accepted_answer_id | int(11) | YES | | NULL | |
| parent_id | int(11) | YES | MUL | NULL | |
| score | int(11) | YES | | NULL | |
| view_count | int(11) | YES | | NULL | |
| body_text | text | YES | | NULL | |
| owner_id | int(11) | NO | | NULL | |
| last_editor_user_id | int(11) | YES | | NULL | |
| last_editor_display_name | varchar(40) | YES | | NULL | |
| last_edit_date | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| last_activity_date | timestamp | NO | | 0000-00-00 00:00:00 | |
| title | varchar(256) | NO | | NULL | |
| answer_count | int(11) | NO | | NULL | |
| comment_count | int(11) | NO | | NULL | |
| favorite_count | int(11) | NO | | NULL | |
| created | timestamp | NO | | 0000-00-00 00:00:00 | |
+--------------------------+--------------+------+-----+---------------------+-----------------------------+
I have setup last_edit_date as not null column.
Based on the answer provided I made the change but error still remains the same:
def convert_to_mysql_time(date='1973-01-01T01:01:01.000')
DateTime.parse(date).to_time.strftime("%F %T")
end
def on_start_element(element, attributes)
if element == 'row'
#post_st.execute(attributes['Id'], attributes['PostTypeId'], attributes['AcceptedAnswerId'], attributes['ParentId'], attributes['Score'], attributes['ViewCount'],
attributes['Body'], attributes['OwnerUserId'] == nil ? -1 : attributes['OwnerUserId'], attributes['LastEditorUserId'], attributes['LastEditorDisplayName'],
convert_to_mysql_time(attributes['LastEditDate']), DateTime.parse(attributes['LastActivityDate']).to_time.strftime("%F %T"), attributes['Title'] == nil ? '' : attributes['Title'],
attributes['AnswerCount'] == nil ? 0 : attributes['AnswerCount'], attributes['CommentCount'] == nil ? 0 : attributes['CommentCount'],
attributes['FavoriteCount'] == nil ? 0 : attributes['FavoriteCount'], DateTime.parse(attributes['CreationDate']).to_time.strftime("%F %T"))
post_id = attributes['Id']
Here is the error:
/ruby/1.9.2/ubuntuamd1/lib/ruby/1.9.1/date/format.rb:1031:in `dup': can't dup NilClass (TypeError)
from /soft/ruby/1.9.2/ubuntuamd1/lib/ruby/1.9.1/date/format.rb:1031:in `_parse'
from /soft/ruby/1.9.2/ubuntuamd1/lib/ruby/1.9.1/date.rb:1732:in `parse'
from load.rb:102:in `convert_to_mysql_time'
from load.rb:109:in `on_start_element'
from load.rb:169:in `parse'
from load.rb:169:in `<main>'

Answer 1

I would, write a method that converts String dates to MySQL dates, and supply it a default value if the nil is supplied to the method, e.g:
def convert_to_my_sql_date(date)
date = '1973-01-01T01:01:01.000' if (date.empty? rescue true) #was added since empty string gets supplied as an argument, and the rescue to make arguments that do not respond to empty? take a default date
DateTime.parse(date).to_time.strftime("%F %T")
end
So when the date is nil it uses the default, then you can now use as below in your method:
convert_to_my_sql_date(attributes['LastEditDate'])