Whats the best way to get JSON flat files into SQL Server using SSIS?
Currently I've tried parsing the data in a script component, but with amount of JSON files I'm parsing (around 120 at a time) it takes upward of 15 minutes to get the data in. I also don't consider this very practical.
Is there a way to combine the powers of SSIS and the OPENJSON command in SQL server? I'm running SQL server 2016 so I'm trying to leverage that command in the hopes that it works faster.
Also, I have no problem getting the JSON data in without losing format.
Looks like this:
Is there a way for me to leverage that and get JSON format into a more Normalized format.
This guy has an example to splitting a JSON string that is in a column that would be a good easy basis.
SSIS Data flow task runs by itself but not as part of package
You would want a class referencing a class if you have subclasses. Kind of like an order class reference a line item class.
In that example, you would have a DF on foreach order and within that a foreach lineitem including the order ID.
I had a good example with Survey Monkey but I can't find it right now.
I actually didn't use data flows with that example and just directly loaded from C#.
Here is the survey monkey class structure i referenced above:
Trouble using all members in a class. Why can I only use the list?
Good luck.
Actually figured this out.
I bring the files in one at a time, with all the JSON text in a single row.
From there I can use the OPENJSON command in SQL Server 2016.
Related
Just gonna start with saying, I sure do know how to pick a challenge (based on my skillsets or the lack of).
I'm trying to figure out how to generate a true/false style table through PowerAutomate from a PowerBI output (Exciting, right?!).
The data is a report of Active Directory users and their group memberships. A Sample of the source data looks like this;
The desired output would look something like this;
How the data is being generated through PowerAutomate and PowerBI;
I'm trying to research JSON Schemas and how that all is supposed to work, but I am just really not sure where to start. Finding an example that matches well enough to replicate into my use case is turning out to be more difficult. Assuming this is a function JSON can complete.
I'm trying out these sites, its helping me learn but is probably not going to solve my issue;
https://odileeds.github.io/JSONSchema/
https://json-schema.org/blog/posts/applicability-json-schema-fundamentals-part-1
Any help would be very much appreciated. Not always looking for the direct answer to the problem, if documentation exists that would push me in the right direction.
There is a new DAX function added recently called TOJSON that might solve your problems, and I believe the imagined use-cases for this function includes your current challenge.
Here is what I would do:
Prepare your table exactly as in your desired output, in Power BI
Lift the entire query used to generate the table using Performance Analyzer
In DAX Studio, paste the query you copied and isolate the final output table - wrap this in TOJSON and verify your results
Use this query in Power Automate - should give you the JSON response you are after directly
If you need to parse the JSON, you can copy the output from DAX Studio as basis for the schema in Power Automate
I have a folder with a lot of data files in. I want to be able to loop through the files, look at the headers and sort them into folders if they have the same headers. Is that possible to do in SSIS? If so would anyone be able point me the direction of how to do this?
I am going to try and explain this as best I can without writing a book as this a multi stepped process that isn't too complex but, might be hard to explain with just test. My apologies but I do not have access to ssdt at the moment so I can not provide images to aid here.
I would use the TextFieldParser class in the VisualBasics.dll. in a script task. This will allow you to read the header from file into a string array. You can then build the string array into a delimited column and load an object variable with a datatable that has been populated with two columns. The first column being the filename and the second being the delimiter headers.
Once you have this variable you can load a sql table with this information. (optional to skip if you want to load the columns directly into sql as you read them. your call)
Once you have your sql table you can create an enumerator for that dataset based on the unique headers column.
Then use a foreach loop task with script task to enumerate thru the unique header sets. Use a sql task to assign the file names that belong to the unique header set.
Within the script loop thru the returned file names and apply the necessary logic to move the files to there respective folders.
This is sort of a high level overview as I am assuming you are familiar enough with SSIS to understand the steps necessary to complete each step. If not then I would be able to elaborate later in the day when I am able to get to my SSIS rig.
Using coldfusion I am using a cvs file and then compiling multiple txt files form that depending on the value of each row in the csv file
I would like to do a SELECT DISTINCT on the csv for the Bank Name column if that is possible, I want to get the distinct values in that column withing the csv. I then also want to count how many rows there are for that distinct value (how many times it appears in the csv file). Finally I want to get the SUM of the Amount column for that distinct Bank Name
I am not really sure how to go about this and would appreciate any input and thank you in advance!
Read your csv file using cfhttp. The name attribute creates a query variable of the file contents which enables you to use query of queries. Details are in the documentation of the cfhttp tag.
You could try using a datasource proxy with a text driver as is described in this post on CSV files and ColdFusion. Because of the 64bit and 32bit ODBC issues you may also need to refer to this post to get such a DSN installed on a modern CF installation. Note that the second post will work through CF 9, but I've not tested the technique on CF 10 or CF 11 (it's a pretty old technique).
I'm not recommending either approach but assuming you could get it working it would give you an easy way to use Q of a Q and get distinct values. I'm not sure if either one of them is any better than Ben's way of doing it. However, you can borrow his CFC and simply pass in your columns and data. I'm not sure I understand how that is more work than writing filtering code.
i trying to figure out if its possible to pre-process a CSV file in SSIS before importing the Data into SQL.
I currently receive a file that contains 8 tables with different structures in one flat file.
the Tables are identified by a row with the Table name in it encapsulated by Square Brackets i.e. [DOL_PROD]
the the data is underneath in standard CSV format. Headers first and then the data.
the tables are split by a blank line and the process repeats for the next 7 tables.
[DOL_CONSUME]
TP Ref,Item Code,Description,Qty,Serial,Consume_Ref
12345,abc,xxxxxxxxx,4,123456789,abc
[DOL_ENGPD]
TP Ref,EquipLoc,BackClyLoc,EngineerCom,Changed,NewName
is it posible to split it out into seperate CSV files? or Process it in a loop?
i would really like to be able to perform this all with SSIS automatically.
Kind Regards,
Adam
You can't do that by flat file source and connection manager alone.
There are two ways to achieve your goal:
You can use Script Component as source of the rows and to process the files, and then you'd do whatever you want with a file programatically.
The other way, is to read your flat file treating every row as a single column (i.e. without specifying delimiter), and then, via Data Flow Transformations, you'd be splitting rows, recognizing table names, splitting flows and so on.
I'd strongly advise you to use Script Component, even if you'd have to learn .NET first, because the second option will be a nightmare :). I'd use Flat File Source to extract lines from file as single column, and thet work it in Script Component, rather than reading a "raw" file directly.
Here's a resource that should get you started: http://furrukhbaig.wordpress.com/2012/02/28/processing-large-poorly-formatted-text-file-with-ssis-9/
I am using mysql together with EF4 and VS2010. I am using model first and uses the Generate database from model function to create the database.
Problem is that the binary properties in the model is converted to varbinary(100). I need them to be converted to MEDIUMBLOB.
I know I can change this manually but it would be really great if this could be done automaticly.
Is one-time mapping change automatically enough? ;-)
You can do this in model designer, just select Table mappings from Entity RMB menu, .
EDIT:
That is weird, but it looks like actually Table Mappings does not allow to change mapping types.
Anyway, what you can do is to change T4 template to generate SQL. To do this, find file:
VS_BASE_DIR\Common7\IDE\Extensions\Microsoft\Entity Framework Tools\DBGenSSDLToSQL10.tt
and modify line 165 to return proper SQL Type name instead of the one returned by ToStoreType() extension method of EdmProperty. Unfortunately I didn't find a way to modify it directly.
Of Course this will modify settings for all further projects. If that is not desired, you can make a copy of original template and chose right one through EF Model Properties\DDL Generation Template in every project.