Comparing ActiveRecord results and only using varied ones - mysql

Question

I'm currently needing to sync data via a redis slave to a remote site. The data lives in MySQL. To do this, I've devised a sync script similar to this:
MyTable
.select("id, first_name, status")
.find_each do |user|
STDOUT.write(gen_redis_proto("SET", "users:#{user.id}",user.to_json))
end
This works perfectly. I pipe this to redis-cli --pipe (as per https://www.redis.io/topics/mass-insert) and it inserts to the local master and syncs to the remote slave.
Unfortunately I have several thousand rows, making this sync quite large. I'd like to only sync rows that have changed, however there's no "last_modified" or similar value available in the table.
The above code runs in a loop with a sleep between runs, so I can store the previous resultset and make a comparison, but I can't work out an efficient way to do this. I'm thinking something similar to the below pseudocode:
lines = [
"{\"id\":123,\"first_name\":\"Jimmy\",\"status\":1}",
"{\"id\":456,\"first_name\":\"John\",\"status\":2}",
"{\"id\":789,\"first_name\":\"James\",\"status\":2}"
]
previous_lines = [
"{\"id\":123,\"first_name\":\"Jimmy\",\"status\":2}",
"{\"id\":456,\"first_name\":\"John\",\"status\":3}",
"{\"id\":789,\"first_name\":\"James\",\"status\":2}"
]
varied_lines = diff(lines, previous_lines) # returns something like [0,1]
varied_lines.each do |line|
this_line = line.to_a
STDOUT.write(gen_redis_proto("SET", "users:#{this_line.id}",line))
end
I suspect too much manipulation of data or comparison will come with a performance overhead, and I'm also unsure of the best way to diff this data to get results.

Answer 1

Why you don't need it, the rickshaw gem lets you add hashing onto strings easily. I'd probably do something with storing & comparing hash values which would take up less space. example.
require 'rickshaw'
def diff(current, previous)
current.select.each_with_index do |line, idx|
previous_line = previous[idx]
line.to_sha1 == previous_line.to_sha1
end
end
lines = [
"{\"id\":123,\"first_name\":\"Jimmy\",\"status\":1}",
"{\"id\":456,\"first_name\":\"John\",\"status\":2}",
"{\"id\":789,\"first_name\":\"James\",\"status\":2}"
]
previous_lines = [
"{\"id\":123,\"first_name\":\"Jimmy\",\"status\":2}",
"{\"id\":456,\"first_name\":\"John\",\"status\":3}",
"{\"id\":789,\"first_name\":\"James\",\"status\":2}"
]
varied_lines = diff(lines, previous_lines)
varied_lines.each do |line|
this_line = line.to_a
STDOUT.write(gen_redis_proto("SET", "users:#{this_line.id}",line))
end