Connection forcibly closed when copying from MySQL to MongoDB - mysql

Question

I've written a script that transforms my data from MySQL to MongoDB. During the handling of a table with 4,000,000 rows I got (when it was almost done):
Traceback (most recent call last):
File "C:\Python32\lib\site-packages\pymongo\connection.py", line 822, in _send_message
sock_info.sock.sendall(data)
socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "kolibri_to_mongo.py", line 94, in <module>
coll.update(..., upsert=True)
File "C:\Python32\lib\site-packages\pymongo\collection.py", line 411, in update
_check_keys, self.__uuid_subtype), safe)
File "C:\Python32\lib\site-packages\pymongo\connection.py", line 837, in _send_message
raise AutoReconnect(str(e))
pymongo.errors.AutoReconnect: [Errno 10054] An existing connection was forcibly closed by the remote host
Exception mysql.connector.errors.InternalError: InternalError() in <bound method SqlConn.__del__ of SQLConn(?)> ignored
Is that a PyMongo Error or an SQL Error? Can I check for any limits (size or timeout) on the MySQL or MongoDB side? Or did just someone kill my query?
EDIT: I've notice that now I cannot connect to the MongoDB anymore with a timeout error :( Are there any limits in MongoDB that need to be changed is it more likely to be another IT/Hardware problem?

Answer 1

That error is coming from MySQL, however, as to why is a bit of an unknown. The error appears to indicate that the remote end, in this instance, MongoDB closed the connection.
I would recommend looking in the pymongo logs to see if there's any further information there.
I would also run tcpdump on the MySQL side to see if the MongoDB server is rejected the connection attempts with a RST or if the SYNs are leaving the MySQL server and simply being ignored (for the latter, you should see the syns spaced out accordingly based on the retransmission timer, e.g. Attempt 1 # 0s, 2 # +3s, 3 # +9s, 4 # +21s).
On the MongoDB server, does netstat -an | grep LIST or sudo lsof -c mongod show that MongoDB is still listening on port 27017 (assuming you haven't changed the default)?
With regard to MongoDB connections errors, the classic case is where ulimit settings are too low and the server runs out of file descriptors. Here are two good links for you to read:
Production Notes
Ulimit

Answer 2

This is a disconnect on the MySQL server - I wonder if you are hitting a query timeout or a deadlock.
Do you have any slow queries or long queries or are you getting any errors in your MySQL logs?