Is it possible to use SFTP inside a GCP Cloud function? [duplicate] - google-cloud-functions

Question

I'm trying to send a file from a bucket to a FTP server using pysftp.
For this I'm using Google cloud functions with python 3.7
I have tried with many different ways, but always with an error.
1.- Downloading the file as string: in this example, to avoiding error with the file content, I will create a file which contains "test string" instead of using what is inside of the bucket file
def sftp_handler():
myHostname = "hotsname.com"
myUsername = "username"
myPassword = "pass"
try:
keydata = b"""AAAAB..."""
key = paramiko.RSAKey(data=decodebytes(keydata))
cnopts = pysftp.CnOpts()
cnopts.hostkeys.add('hotname.com', 'ssh-rsa', key)
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword,port=22,cnopts=cnopts) as sftp:
print ("Connection succesfully stablished ... ")
remoteFilePath = '/dir1/dir2/dir3/'
sftp.putfo(StringIO('test string'), os.path.join(remoteFilePath, "OC.csv"))
# connection closed automatically at the end of the with-block
except:
print("Unexpected error in sftp_handler:")
traceback.print_exc()
The error output (stackdriver log )
Traceback (most recent call last): E undefined
File "/user_code/main.py", line 36, in sftp_handler E
sftp.putfo(StringIO('xml string'), os.path.join(remoteFilePath, "OC.csv")) E
File "/env/local/lib/python3.7/site-packages/pysftp/__init__.py", line 474, in putfo E
callback=callback, confirm=confirm) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 714, in putfo E
with self.file(remotepath, "wb") as fr: E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 372, in open E
t, msg = self._request(CMD_OPEN, filename, imode, attrblock) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 813, in _request E
return self._read_response(num) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 865, in _read_response E
self._convert_status(msg) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 898, in _convert_status E
raise IOError(text)
OSError E
2.- Using a temp file in which I will write the content of the file from the bucket
<adding this to the previous code>
with tempfile.NamedTemporaryFile(suffix='.csv', prefix=os.path.basename(__file__)) as tf:
tf.write(b'Hello world!')
remoteFilePath = '/dir1/dir2/dir3/'
sftp.put(os.path.dirname(tf.name), os.path.join(remoteFilePath, "OC.csv"))
The error log:
Traceback (most recent call last): E
File "/user_code/main.py", line 40, in sftp_handler E
sftp.put(os.path.dirname(tf.name), os.path.join(remoteFilePath, "OC.csv")) E
File "/env/local/lib/python3.7/site-packages/pysftp/__init__.py", line 364, in put E
confirm=confirm) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 758, in put E
with open(localpath, "rb") as fl: E
IsADirectoryError: [Errno 21] Is a directory: '/tmp' E
3.- As Martin suggested in his answer:
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword,port=22,cnopts=cnopts) as sftp:
print ("Connection succesfully stablished ... ")
remoteFilePath = '/dir1/dir2/dir3/'
sftp.putfo(StringIO('test string'), remoteFilePath + "OC.csv")
I get the following error:
Traceback (most recent call last): E
File "/user_code/main.py", line 36, in sftp_handler E
sftp.putfo(StringIO('test string'), remoteFilePath + "OC.csv") E
File "/env/local/lib/python3.7/site-packages/pysftp/__init__.py", line 474, in putfo E
callback=callback, confirm=confirm) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 714, in putfo E
with self.file(remotepath, "wb") as fr: E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 372, in open E
t, msg = self._request(CMD_OPEN, filename, imode, attrblock) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 813, in _request E
return self._read_response(num) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 865, in _read_response E
self._convert_status(msg) E
File "/env/local/lib/python3.7/site-packages/paramiko/sftp_client.py", line 898, in _convert_status E
raise IOError(text) E
OSError E
4.- Downloading the file from the bucket, create a file while executing the cloud functions and send this file to the sftp (Not implemented yet as I'm not sure if it will work creating a file in a Cloud Function)
How I could do this?
Why is failing?
Note: I also tried using FileZilla and I also have an error, so this should be something related the FTP server:
09:00:46 Status: Connecting to sftp-staging.messagesnetwork.com...
09:01:06 Status: Connected to sftp-staging.messagesnetwork.com
09:01:06 Status: Retrieving directory listing...
09:01:06 Status: Listing directory /
09:01:07 Status: Directory listing of "/" successful
09:01:11 Status: Retrieving directory listing of "/folder1"...
09:01:37 Command: cd "folder1"
09:01:37 Response: New directory is: "/folder1"
09:01:37 Command: ls
09:01:37 Error: Connection timed out after 20 seconds of inactivity
09:01:38 Error: Failed to retrieve directory listing
09:01:38 Status: Disconnected from server
09:01:38 Status: Connecting to sftp-staging.messagesnetwork.com...
09:02:04 Status: Connected to sftp-staging.messagesnetwork.com
09:02:09 Status: Retrieving directory listing of "/folder1"...
09:02:10 Status: Listing directory /folder1
09:02:10 Status: Directory listing of "/folder1" successful
09:02:12 Status: Retrieving directory listing of "/folder1/folder2"...
09:02:21 Status: Listing directory /folder1/folder2
09:02:23 Status: Directory listing of "/folder1/folder2" successful
09:02:24 Status: Retrieving directory listing of "/folder1/folder2/folder3"...
09:02:25 Status: Listing directory /folder1/folder2/folder3
09:02:26 Status: Directory listing of "/folder1/folder2/folder3" successful
09:02:30 Status: Connecting to sftp-staging.messagesnetwork.com...
09:02:41 Status: Connected to sftp-staging.messagesnetwork.com
09:02:45 Status: Starting upload of <...>\Desktop\folder2cf.txt
09:02:59 Command: cd "/folder1/folder2/folder3"
09:02:59 Response: New directory is: "/folder1/folder2/folder3"
09:02:59 Command: put "<...>\Desktop\folder2cf.txt" "folder2cf.txt"
09:02:59 Error: /folder1/folder2/folder3/folder2cf.txt: open for write: failure
09:02:59 Error: File transfer failed
09:02:59 Status: Starting upload of <...>\Desktop\folder2cf.txt
09:02:59 Status: Retrieving directory listing of "/folder1/folder2/folder3"...
09:02:59 Status: Listing directory /folder1/folder2/folder3
09:03:11 Command: put "<...>\Desktop\folder2cf.txt" "folder2cf.txt"
09:03:11 Error: /folder1/folder2/folder3/folder2cf.txt: open for write: failure
09:03:11 Error: File transfer failed
09:03:11 Status: Starting upload of <...>\Desktop\folder2cf.txt
09:03:11 Status: Retrieving directory listing of "/folder1/folder2/folder3"...
09:03:11 Status: Listing directory /folder1/folder2/folder3
09:03:34 Command: put "<...>\Desktop\folder2cf.txt" "folder2cf.txt"
09:03:34 Error: /folder1/folder2/folder3/folder2cf.txt: open for write: failure
09:03:34 Error: File transfer failed
09:03:34 Status: Retrieving directory listing of "/folder1/folder2/folder3"...
09:03:39 Status: Listing directory /folder1/folder2/folder3
09:03:44 Status: Directory listing of "/folder1/folder2/folder3" successful

Answer 1

sftp.putfo(StringIO('test string'), os.path.join(remoteFilePath, "OC.csv"))
I cannot say atm, if this is the real problem, but you should not use path.join for SFTP paths. SFTP always uses forward slash. path.join uses a separator of the local operating system. It may be different (particularly on Windows).
The code should be:
remoteFilePath = '/dir1/dir2/dir3/'
sftp.putfo(StringIO('test string'), remoteFilePath + "OC.csv")
sftp.put(os.path.dirname(tf.name), os.path.join(remoteFilePath, "OC.csv"))
Use of the path.dirname makes so sense here. You want to upload a file, so you have to give the path to a file to upload, not a path to its containing folder.
And there's the same problem with path.join as previously.