When using Data Connection, can I ingest files into Foundry from an SFTP (SSH File Transfer Protocol, not FTPS) source?
Data Connection doesn't support STFP out of the box. However, there is a custom source.
Ask your Palantir team to install it.
Related
Is it possible for a Data Connection agent to connect to NFS in order to transfer data?
There is no out of the box support for connecting directly, however, you could mount the NFS on the agent host (https://linuxize.com/post/how-to-mount-an-nfs-share-in-linux/), and then just use a standard Directory source to ingest the data.
I am looking to create a SSIS package which downloads a file from an Azure DevOps GIT repository.
The file is a excel spread sheet.
The SSIS package should download this file in a local directory which can then be used for further processing.
Is this possible to achieve using SSIS?
Is this possible to achieve using SSIS?
I'm afraid, no, this could not be achieved via SSIS until now. Because what is available in Azure devops is build and deploy SSIS package. We haven't support to run SSIS package in Azure devops pipeline at this time.
At present, you can make use Azure Data Factory, which is the one service of Azure. See this doc.
But also, as far as I know, it does not support activity of copy file via SSIS. And if you want the excel file can be further processing with SSIS, you may consider to use Azure file copy task to copy the excel file to Azure blob:
You can do just about anything you want in Azure Devops using either the Powershell release template or the Run Powershell on Remote Machines release template. The only catch here is setting up WinRM on the remote servers, opening ports/firewall settings, using X509 self-signed certs if you want to do this over SSL, WinRM listeners, setting up trusted hosts, etc. etc. It took me two months to finally get WinRM setup and working correctly from Azure Devops (in my case that included setting up a special security group policy to allow the WinRM services to run unimpeded on the remote machines joined to the domain). Once you have WinRM working though, you can have Azure Devops drive anything that you could script with Powershell so the effort was worth it for me. If you embark on this adventure, take the time to write some powershell test scripts that call Invoke-command to test all of the WinRM security features I mention above, this will save you a lot of time with troubleshooting a remote connection over WinRM.
We have to read CSV file from SFTP server/folder and with the help of Kafka I have to push this data into Salesforce Sobject .Similarly we have to read any Sboject Data from salesforce and convert this data into into CSV file and send to SFTP server.
Can you please give me some idea about that ?How i can achieve this?
i did not check it very careful but maybe you can have a reading for this ,if you could query data from salesforce then you can change it to any formats.
Salesforce Connector (Source and Sink) for Confluent Platform — Confluent Platform
Salesforce Platform Events Sink Connector for Confluent Platform — Confluent Platform
I am using ssis to upload multiple files to azure blob storage. Requirement is when any of the file upload fail we need to rollback transactions. I have tried the transaction option in ssis but so far i am not able to rollback data from blob storage.
Has anyone tried the rollback option in azure blob storage? Please do let me know your thoughts on this.
Thanks
Vidya
There are two options. If you are importing .csv or AVRO files, you can use the SQL Server Integration Services Feature Pack for Azure installed where:
The Azure Blob Source component enables an SSIS package to read data from an Azure blob. The supported file formats are: CSV and AVRO.
If you are not working with these file formats, then the solution offered in this SO thread: How to transaction rollback in ssis? with the use of Sequence Containers.
The SSIS Feature Pack for Azure contains connectors for various other data services and might be useful if you are going to be consuming additional services in the future.
I am working on using SSIS FTP task to uploading some files to sftp://apollo.xxx.xxx.com automatically.
I can do that successfully by manually specifying sever name, server port, user name and password in FileZilla.
However, when I setup in the FTP connection manager, the connection fails (could be the issue of SFTP?).
I am just wondering if anyone could provide me some tips?
The native FTP task is just FTP. It does not support sFTP.
As you are not allowed to use third party software, you will be unable to fulfill your request. Based on this question, there does not appear to be a base sftp module in the .NET framework How do I upload a file to an SFTP server in C# / .NET?