Not sure how to ask this but here goes. I have a collection of 500+ JSON files that I need to import into PowerBI. Each JSON has four different levels of information that I need to parse out. I converted the JSON top-level info into a table and transposed it so I had one row like the attached screenshot.
enter image description here
My first question is: can I easily add the filename to the JSON record? I would like to use the filename as a unique key in later queries.
Thanks!
It looks like you may be connecting to each JSON file individually? If I'm correct, assuming all the JSON files can be in a single folder, you can use the "Folder" connection. That then allows you to right-click on the original folder query and choose "reference" to then create various transformations for each JSON file, and it includes the file name.
Related details:
https://powerbi.tips/2016/06/loading-data-from-folder/
https://learn.microsoft.com/en-us/power-bi/guidance/power-query-referenced-queries
Hoping that helps!
Related
I'm trying to map a JSON document (sensor data) into a more meaningful representation using Mapping Dataflows. However, hard time getting this to work and would really appreciate some insight/recommendations on how to solve the following:
The input is
What I would like to end up with is the following:
Any pointers as to how this can be implemented are more than welcome.
This can be accomplished using the Copy activity and then split function in Derived Column transformation in Azure Data Factory.
Use the copy activity to read the JSON file as source and in sink, use SQL database to store the data as table. In Mapping tab, Import the schema and map the JSON records to the corresponding column names. Refer this third-part tutorial for guidance - https://sqlkover.com/dynamically-map-json-to-sql-in-azure-data-factory/
Finally, use the Data Flow activity and choose the SQL table as source now which you have used as sink above.
Select the Derived Column transformation.
Use split function.
Add the column which will take the split values which you want to split as shown below.
Use split(<column_name_to_split>, '_') function to split the column on with _ delimiter. Change <column_name_to_split> to the name of column you cant to split. Refer image below.
Preview the data to check the result.
In my .csv file, I have the list of firstName and lastName. I want to use first entry in one feature file and the 2nd entry in another feature file. Is it possible to do that?
See if this helps: https://github.com/intuit/karate#calling-other-feature-files - note that you can "pass data" to other feature files.
And note that when you read a CSV file, you get JSON: https://github.com/intuit/karate#csv-files
If that does not help - then the answer is - no it is not possible to do that !
I have about a hundred JSON files of data that I would like to be able to manipulate in Excel. The reason why there are so many files is that the API I pulled from limits responses to 50 items per request, so I chained 100 requests together in Postman and each request generated its own file.
The layout of each file is as follows:
{
"href": "dsjdsjds.com",
"total": 4293,
"next": "sdsadsads.com",
"prev": "dsjdjsdj.com",
"limit": 50,
"offset": 50,
"itemSummaries": [...]
}
Pretty much all of the data that I want lies inside the itemSummaries class.
I'm pretty new to this and not sure if the optimal way would be to use a Python script, or if there was a way to use VBA or something. I was thinking that I'd need to combine all of the data into a single file first, but I don't know how to do that either. I appreciate the help!
Here is what I did for similar situation where I had to import multiple JSON files, all with same structure.
Use Get & Transform in Data Ribbon to import the JSON file as text.
The Power Query will recognize this as JSON. Edit the result in Power
Query window and expand/transform the imported data until you can
show in tabular form.
You can then convert these manual sequence of
steps into Custom Function. See here for details -
https://www.poweredsolutions.co/2019/02/19/parameters-and-functions-in-power-bi-power-query-custom-functions/
Go back to Excel and this time instead of importing the JSON file, import the folder where all these JSON are available and apply your custom function on the individual JSON files to produce a consolidated table.
I found this article.
Import from JSON
Select Data > Get Data > From File > From Folder. The Browse dialog box appears.
Locate the folder containing the files you want to combine.
Note The message “No items match your search” means Power Query has found a folder and it’s displayed in the Folder name box. The files you want to combine are still in the folder, but just not visible.
A list of the files in the folder appears in the dialog box. Verify that all the files you want are listed.
Select one of the commands at the bottom of the dialog box, for example Combine > Combine & Transform. There are additional commands discussed in the section About all those commands.
The Power Query Editor appears.
The Value column is a structured List column. Select the Expand icon, and then select Expand to New rows.
The Value column is now a structured Record column. Select the Expand icon. A drop-down dialog box appears.
Keep all the columns selected. You may want to clear the Use original column name as a prefix check box. Select OK.
Select all the columns that contain data values. Select Home, the arrow next to Remove Columns, and then select Remove Other Columns.
Select Home > Close & Load.
Result
Power Query automatically creates a query to consolidate the data from each file into a worksheet. The query steps and columns created depend on which command you choose. For more information, see the section, About all those queries.
Let's say I have 100 scanned Tif files. I also have a CSV of the metadata for those 100 Tif files. Each file is named with its unique identifier, which is also column 1 of the csv.
First: How do I find a map that tells me what columns should be named what, in order to stay within the IPTC standard using XMP? (I've googled for most of the day and have found nothing)
Second: How can I merge the metadata in the CSV to each corresponding image?
I'm basically creating a spreadsheet with all 50,000 images in an archival collection, and plan to use the CSV to create the metadata for the images once they're scanned.
Thanks!
To know where to put your metadata, I'd suggest looking at the IPTC Photo Metadata Standard page. Without knowing more about your data, it's hard for someone else to say what data should go where.
As for embedding your data into your files from a CSV file, I'd suggest exiftool. Change the header of each column to the name of the TAG to write to and make the first column the path/filename of each file, your command would be as simple as
exiftool -csv=file.csv /path/to/files
See exiftool FAQ #26 for more details.
SSIS
This is general question which i am going to ask and might possible to have multiple answer.
Techie, please share with me the best solution if familiar with scenario.
i have 100 customer and they share file for us to load into our database.
and after compution and the output file need to be saved in Output folder.
the name of the output file should be customer_name.txt where customer_name
is coming from a column available in input file.
Can anyone please help me how should i design this in SSIS to achieve my Goal
At first, use a Foreach Loop Container for every file.
For each one, store data into your importTable in database with the fileName or customerName.
Then, when you need to get output file:
Get the nameFile or userName with a query and save it into a ssis variable.
Create your file with the 'DestinationVariable' as 'User::yourVariableName'
Add the data from your database and then use a simple script to move it into output folder
Example