I have an XML Source task which requires a XSD to generate my data flow work from my XML files in SSIS. How can I embed the XSD file within the SLN/DTSX project and have it be referenced correctly? It would seem the best case would be to have the XSD source inside a variable, but I dont see where I can do that given the XML Source data task
Using Project Deployment model it is possible to get the miscellaneous files bundled with the deployment. However I want to use the Project Deployment model. That model does not allow for files other than the ispac file. To get around the problem using this following post as reference SSIS dynamic xsd source file
Effectively you store the XSD within a available sql server database as a XML schema, use a dataflow to query for it, and store it in a RAW FILE. Then save the RAW file to disk for future use. Very Effective
Related
Automatic JSON Files upload to Blob Storage.
Description:
We have a SSIS job which will generate JSON files with data at a server path. We are manually copying the JSON files and dropping them in BLOB storage in order to trigger our logic app.
Now, Could anyone help to provide information on how we can automate the process of copying JSON files to BLOB automatically? ( Like do we have any approach or code to copy the JSON files at a specific time and copy those JSON files in BLOB )
The solution is to listen to the file system change at your server path, then to use Azure Storage SDK to upload these files which be triggered by the file changed event.
As reference, here are some resources about the API or SO threads of file changes listener in different languages, because I don't know what language you want to use.
C# FileSystemWatcher Class
Python How do I watch a file for changes?
Node.js Observe file changes with node.js
For other languages, I think you can easily get their solution by searching. And to upload files to Azure Storage, you just need to refer to Azure offical getstarted tutorials in dfferent languages to write your code.
I have created a SSIS package that reads from a CSV file (using the Flat file connection manager) and loads records into a database. I have deployed it on Azure data factory pipeline and I need to give the path of the CSV file as a parameter. I have created a azure storage account and uploaded the source file there as shown below.
Can I just give the URL of the source file for the Import file in the SSIS package settings as shown below? I tried it but it is currently throwing 2906 error. I am new to Azure - appreciate any help here.
First, you said Excel and then you said CSV. Those are two different formats. But since you mention the flat file connection manager, I'm going to assume you meant CSV. If not, let me know and I'll update my answer.
I think you will need to install the SSIS Feature Pack for Azure and use the Azure Storage Connection Manager. You can then use the Azure Blob Source in your data flow task (it supports CSV files). When you add the blob source, the GUI should help you create the new connection manager. There is a tutorial on MS SQL Tips that shows each step. It's a couple years old, but I don't think much has changed.
As a side thought, is there a reason you chose to use SSIS over native ADF V2? It does a nice job of copying data from blob storage to a database.
I did a conversion program to change the object structure of a DWFx file, and it works fine. What I did was to open the DWFx file as a zip archive, parse the internal XML files, and reorganize them, creating new parent nodes when needed.
But what doesn't work is changing the names of these nodes. When I open the file in any Autodesk viewer (the offline Design Review program and the online Viewer are the ones I tested), the tree structure is changed as it should, but the parent node names are not. In fact, the nodes that already existed keep their old names, and the new ones are called Object XXXX. The child nodes (actual objects) have their names changed correctly.
I tried to search in every readable (text) file inside the DWFx, but none of them hold any other reference to these nodes. I didn't open binary files, like W3D files, which probably hold the geometry.
Does anyone have any experience in creating or altering DWFx files? Do I need to change anything else besides the 'label' tag in the Presentation XML file?
Instead of manipulating the contents of DWFx files manually, consider using one of the Autodesk Forge services: Design Automation. The APIs allow you to run AutoCAD "in the cloud". You could theoretically load your DWFx file there, update the structure/names, and generate an updated DWFx file. Here's an example of how the service can be used to generate PDFs out of DWG files: https://forge.autodesk.com/en/docs/design-automation/v2/tutorials/convert-dwg-to-pdf.
My workplace has a whole bunch of unannotated .zip files that need to be uploaded to the new file server (Windows). I've used perl to parse through through the excel files within the .zip files to create an annotation.txt file for each .zip file that contains information about the .zip file. I have 1000's of zip files and do not want to manually enter in information for each entry if there's a way to automate it. I am proficient in perl and mysql, and wondering if there is any way to utilize my skillsets to port this information into the Microsoft Sharepoint website.
Thank you in advance for any advice or suggestions.
There a many, many ways to meet your requirement.
You could write a event receiver to parse the files once uploaded and set metadata.
A better approach for your use case might be to write a .NET based console application and reference Microsoft.SharePoint.Client and then upload your files using the Client side object model (CSOM) and set the metadata during that process as outlined here: Upload a document to a SharePoint list from Client Side Object Model
There are also REST and ASMX webservices that you could call from a non .NET runtime process.
Plenty of options, pick the one that fits your needs and skills best.
At present I have source XML file, from which I am truncating some values using XSLT file. For this I am using “XML task” in control flow.
Then after getting the resultant file manually I am submitting the resultant XML file to XML source and then loading the data into an SQL table.
If there a better approach to do the same in one go ?
Novice to SSIS. So if possible could someone give example or link to any example to accomplish this
The SSIS Package can contain both control flow as well as one or more data flows. you need to include your both XML Task and Data Flow Task (having XML Source for loading data to sql table). Connect these executable and task in control flow using precedence constraints . This SSIS Package will be automated.
Saving Result from XML Task to file or variable.
In XML Task Editor,
Set SaveOperationResult to True
Set OperationResult\DestinationType either to File Connection or
Variable
Define New Variable or File Connection from Destination property or
select existing one.
In XML Source Editor, select "XML data from Variable" or "XML File Connection" and then specify details in the XML location accordingly.