I am using ssis to upload multiple files to azure blob storage. Requirement is when any of the file upload fail we need to rollback transactions. I have tried the transaction option in ssis but so far i am not able to rollback data from blob storage.
Has anyone tried the rollback option in azure blob storage? Please do let me know your thoughts on this.
Thanks
Vidya
There are two options. If you are importing .csv or AVRO files, you can use the SQL Server Integration Services Feature Pack for Azure installed where:
The Azure Blob Source component enables an SSIS package to read data from an Azure blob. The supported file formats are: CSV and AVRO.
If you are not working with these file formats, then the solution offered in this SO thread: How to transaction rollback in ssis? with the use of Sequence Containers.
The SSIS Feature Pack for Azure contains connectors for various other data services and might be useful if you are going to be consuming additional services in the future.
Related
I would like to copy SQL Server data to Azure Table storage. We can achieve this task using Azure Data Factory. However, I wanted to do this with SSIS package.
I installed feature package for Azure but could not get the Azure Table Storage as destination in SSIS Toolbox.
Is there any other simple way to achieve this?
There are a couple of things that we need to take care of before we walk through building the on-premise SSIS packages:
-Create Azure Subscription
-Create Azure Storage Account
-Create Azure SQL Database
-Install SQL Server Data Tools (SSDT)
-Install Azure Feature Pack for SQL Server Integration Services (SSIS)
-Install SQL Server Management Studio (SSMS)
I think you are missing the Azure Feature Pack for SSIS which includes a handful of components for working with Azure from SSIS.
EDIT 1 :
It seems that Azure Table Storage Source is a third-party component.
EDIT 2 :
You can use SSMS : Right-click on the database and select Tasks-Export Data.
Then choose Destination as Flat File Destination and specify a CSV path as the destination.
Choose "Write a query to specify the data to transfer"
Start Azure Storage Explorer, open the target table which the data would be imported into, and click Import on the toolbar.
Select the file just exported, check and change the data type if necessary for each field.
Click Insert, and then your data will be imported to Azure Storage Table.
You can also verify the data on Azure Portal.
When using Data Connection, can I ingest files into Foundry from an SFTP (SSH File Transfer Protocol, not FTPS) source?
Data Connection doesn't support STFP out of the box. However, there is a custom source.
Ask your Palantir team to install it.
I am looking to create a SSIS package which downloads a file from an Azure DevOps GIT repository.
The file is a excel spread sheet.
The SSIS package should download this file in a local directory which can then be used for further processing.
Is this possible to achieve using SSIS?
Is this possible to achieve using SSIS?
I'm afraid, no, this could not be achieved via SSIS until now. Because what is available in Azure devops is build and deploy SSIS package. We haven't support to run SSIS package in Azure devops pipeline at this time.
At present, you can make use Azure Data Factory, which is the one service of Azure. See this doc.
But also, as far as I know, it does not support activity of copy file via SSIS. And if you want the excel file can be further processing with SSIS, you may consider to use Azure file copy task to copy the excel file to Azure blob:
You can do just about anything you want in Azure Devops using either the Powershell release template or the Run Powershell on Remote Machines release template. The only catch here is setting up WinRM on the remote servers, opening ports/firewall settings, using X509 self-signed certs if you want to do this over SSL, WinRM listeners, setting up trusted hosts, etc. etc. It took me two months to finally get WinRM setup and working correctly from Azure Devops (in my case that included setting up a special security group policy to allow the WinRM services to run unimpeded on the remote machines joined to the domain). Once you have WinRM working though, you can have Azure Devops drive anything that you could script with Powershell so the effort was worth it for me. If you embark on this adventure, take the time to write some powershell test scripts that call Invoke-command to test all of the WinRM security features I mention above, this will save you a lot of time with troubleshooting a remote connection over WinRM.
How to deploy a HDP cluster without HDFS, as I don't want HDFS for storage and will be using in-house in-memory storage system. How can this be done?
HDFS and MapReduce are the main internal part of Hadoop. They comes inbuilt with the package of Hadoop . You can not exclude HDFS while HDP cluster Deployment. You can exlcude other sevices then HDFS but you can not exclude HDFS.
HDFS is an implementation of the Hadoop FileSystem API, which models POSIX file system behavior.
You are probably referring to Object Stores in places, often using the term Blobstore. Hadoop does provide FileSystem client classes for some of these even though they violate many of the requirements. This is why, although Hadoop can read and write data in an object store, the two which Hadoop ships with direct support for — Amazon S3 and OpenStack Swift — cannot be used as direct replacements for HDFS.
For further reading, Object Stores vs Filesystems
I am having one asp.net application in which I used MySQL as database. The MySQL database is having the list of Issues in one of the table. I want to access the Database of our Cloud JIRA and want to insert the issues of MySQL DB into the cloud JIRA DB directly from that asp.net application.
Is there any API available for us to do so? What will be the best approach to fulfill the above requirement?
If you are using JIRA Cloud, you are not able to access the database at all. It's part of the limitation of Cloud. You might be able to use REST API depends on what you are looking to achieve exactly.
However, I would recommend you to find a way to export the issues from your ASP.net Application to CSV file and then restore the CSV file into your Cloud JIRA since JIRA has an ability to create issues from Excel file.
Note that, you may need to contact Atlassian Support for restoring JIRA from CSV file since they do have full control over instances.