I've hundreds of rows in CSV file while contains JSON data like below. Below is a sample of each row.
{"Id":"value","RecordType":"value","CreationTime":"value","Operation":"value"}
I tried to convert the same into CSV as below but no luck as of now.
Expected format of CSV file:
id RecordType CreationTime Operation
value value value value
$properties = #('Id', 'RecordType', 'CreationTime', 'Operation')
(Get-Content -Path-to_CSVfile -Raw | ConvertFrom-Json) |
Select-Object -Property $properties |
Export-Csv -NoTypeInformation -Path $path-to-new-csv-file
If someone has an idea about this please help me. I tried ConvertTo-Json but it's failing with error:
ConvertFrom-Json : Invalid JSON primitive: "id"
Here are the first two rows of CSV data.
{"Id":"ac325bc9-97f0-4b29-8fc4-90b80b945f6c","RecordType":20,"CreationTime":"2019-09-14T08:07:22","Operation":"AnalyzedByExternalApplication","OrganizationId":"f38a5ecd-2813-4862-b11b-ac1d563c806f","UserType":0,"UserKey":"3fee8456-6d20-4794-8219-5a7c381e965f","Workload":"PowerBI","UserId":"abcd#mail.com","ClientIP":"000.000.50.177","UserAgent":"MSOLAP 15.0 Client","Activity":"AnalyzedByExternalApplication","ItemName":"Other","DatasetName":"XYZ Driven Company","ObjectId":"Other","IsSuccess":true,"RequestId":"6836be8e-6e97-4bc9-a838-bf6e7b71e0c8","ActivityId":"7E92AE6A-F548-448D-93A8-6F5736DEA085"}
{"Id":"3a20c8a9-ef44-483a-b9c0-43e10deae9ae","RecordType":20,"CreationTime":"2019-09-14T08:07:20","Operation":"AnalyzedByExternalApplication","OrganizationId":"f38a5ecd-2813-4862-b11b-ac1d563c806f","UserType":0,"UserKey":"3fee8456-6d20-4794-8219-5a7c381e965f","Workload":"PowerBI","UserId":"abcd#mail.com","ClientIP":"000.000.50.177","UserAgent":"MSOLAP 15.0 Client","Activity":"AnalyzedByExternalApplication","ItemName":"Other","DatasetName":"XYZ Driven Company","ObjectId":"Other","IsSuccess":true,"RequestId":"02e5d772-057b-45b6-ae60-22b7fa610f98","ActivityId":"7E92AE6A-F548-448D-93A8-6F5736DEA085"}
I'm looking this data in another CSV file as below. Each value after ":" should insert into CSV as rows.
Id RecordType CreationTime Operation OrganizationId UserType UserKey Workload UserId ClientIP UserAgent Activity ItemName DatasetName ObjectId IsSuccess RequestId ActivityId
ac325bc9-97f0-4b29-8fc4-90b80b945f6c 20 2019-09-14T08:07:22 AnalyzedByExternalApplication f38a5ecd-2813-4862-b11b-ac1d563c806f 0 3fee8456-6d20-4794-8219-5a7c381e965f PowerBI abcd#mail.com 000.000.50.177 MSOLAP 15.0 Client AnalyzedByExternalApplication Other xyz Driven Company Other TRUE 6836be8e-6e97-4bc9-a838-bf6e7b71e0c8 7E92AE6A-F548-448D-93A8-6F5736DEA085
3a20c8a9-ef44-483a-b9c0-43e10deae9ae 20 2019-09-14T08:07:20 AnalyzedByExternalApplication f38a5ecd-2813-4862-b11b-ac1d563c806f 0 3fee8456-6d20-4794-8219-5a7c381e965f PowerBI abcd#mail.com 000.000.50.177 MSOLAP 15.0 Client AnalyzedByExternalApplication Other XYZ Driven Company Other TRUE 02e5d772-057b-45b6-ae60-22b7fa610f98 7E92AE6A-F548-448D-93A8-6F5736DEA085
Correct data from CSV when opened in text editor.
"{""Id"":""ac325bc9-97f0-4b29-8fc4-90b80b945f6c"",""RecordType"":20,""CreationTime"":""2019-09-14T08:07:22"",""Operation"":""AnalyzedByExternalApplication"",""OrganizationId"":""f38a5ecd-2813-4862-b11b-ac1d563abchrf"",""UserType"":0,""UserKey"":""3fee8456-6d20-4794-8219-5a7c38abcdfe"",""Workload"":""Pxyswer"",""UserId"":""abcd#mail.com"",""ClientIP"":""123.456.50.177"",""UserAgent"":""MSOLAP 15.0 Client"",""Activity"":""AnalyzedByExternalApplication"",""ItemName"":""Other"",""DatasetName"":""ABCD Driven Company"",""ObjectId"":""Other"",""IsSuccess"":true,""RequestId"":""6836be8e-6e97-4bc9-a838-bf6e7b71e0c8"",""ActivityId"":""7E92AE6A-F548-448D-93A8-6F5736DEA085""}"
If your input file contains just that single example data row the code you posted will work. If the input file contains multiple statements like that your code will not work, because it'd be invalid JSON data.
Valid JSON:
{"Id":"value","RecordType":"value","CreationTime":"value","Operation":"value"}
Valid JSON:
[
{"Id":"value","RecordType":"value","CreationTime":"value","Operation":"value"},
{"Id":"value","RecordType":"value","CreationTime":"value","Operation":"value"}
]
Invalid JSON:
{"Id":"value","RecordType":"value","CreationTime":"value","Operation":"value"}
{"Id":"value","RecordType":"value","CreationTime":"value","Operation":"value"}
To convert the latter kind of input data you need to convert each row as a separate JSON document:
$properties = 'Id', 'RecordType', 'CreationTime', 'Operation'
Get-Content 'C:\path\to\input.csv' |
ConvertFrom-Json |
Select-Object $properties |
Export-Csv 'C:\path\to\output.csv' -NoType
To export all input fields except particular ones you'd define the properties to exclude rather than the ones to include:
$exclude = 'foo', 'bar'
Get-Content 'C:\path\to\input.csv' |
ConvertFrom-Json |
Select-Object -Properties * -Exclude $exclude |
Export-Csv 'C:\path\to\output.csv' -NoType
Edit:
Apparently your input file is a CSV with only one column and no header, so you can import it via Import-Csv, but you need to specify the column header yourself. Expand the field to get the individual JSON values, then proceed as described above.
$properties = 'Id', 'RecordType', 'CreationTime', 'Operation'
Import-Csv 'C:\path\to\input.csv' -Header foo |
Select-Object -Expand foo |
ConvertFrom-Json |
Select-Object $properties |
Export-Csv 'C:\path\to\output.csv' -NoType
If you want all JSON values exported, simply omit the Select-Object $properties step.
Related
I have a csv in the format below that needs to be converted to json or a table so that I can parse it easily and use the values as arguments later in my code
Directory : \\foo\foofoo\nightly.188\share\name
Name : name.pst
Length : 271360
CreationTime : 6/4/2009 2:42:21 PM
LastWriteTime : 6/9/2011 8:58:50 AM
Directory : \\foo\foofoo\nightly.188\share\name
Name : name2.pst
Length : 71123968
CreationTime : 10/5/2010 2:41:56 PM
LastWriteTime : 8/1/2011 4:08:32 PM
I have tried using Format-List and Format-Table but does not help.
Any inputs would be appreciated.
I would like the output to be in a tabular structure as:
Directory Name Length CreationTime LastWriteTime
\foo\foofoo\nightly.188\share\name name.pst 271360 6/4/2009 2:42:21 PM 8/1/2011 4:08:32 PM
\foo\foofoo\nightly.188\share\name name2.pst 71123968 10/5/2010 2:41:56 PM 8/1/2011 4:08:32 PM
Format-List and Format-table convert your data into a form that's suitable for display on your console, or copying into a text file. This is useful for parsing, but not for passing the data on as arguments. If you just do an import like this:
Import-Csv myfile.csv
Powershell will do an implicit Format-Table or Format-List for you before putting it on your console.
If you want to use the contents as arguments, be aware that the contents of a csv file are transformed into an array of PSCustomObject. Each PSCustomObject has properties with the same name as a the field name provided in the header of the CSV file.
Import-Csv thefile.csv | % {
"The file $($_.name) contains:"
Get-Content $_.name
}
To view CSV data to the console as a table do the following
$sorceFile = "D:\temp\source.csv"
Import-Csv $sorceFile | Format-Table -AutoSize
You can also use the "Out-GridView" command if you want to view the data interactively
$sorceFile = "D:\temp\source.csv"
Import-Csv $sorceFile | Out-GridView
The following converts CSV data to Json data
$sorceFile = "D:\temp\source.csv"
$JsonData = Import-Csv $sorceFile | ConvertTo-Json
$JsonData
This however is just a string. If you are wanting to process the data further from the CSV file then do the following to save to a variable.
$sorceFile = "D:\temp\source.csv"
$CsvData= Import-Csv $sorceFile
# view formatted table in console
$CsvData | format-table
#example data query
$CsvData | Where-Object{$_.Name -eq 'name.pst'}
#example loop
foreach ($row in $CsvData) {
Write-Output "Name: $($row.name) CreationTime: $(get-date $row.CreationTime -Format 'dddd MM/dd/yyyy HH:mm K')"
}
How can I convert the PSobjects returned by Get-Help to strings and export to csv?. I think this is an easy one, but I can't quite get the format I'm looking for. From reading previous answers, I was trying:
Get-Help Get-Content | Select-Object -First 1 -Property #{
Name="temp"
Expression = { $_.Name,$_.Synopsis,$_.syntax,$_.Description }
} | Select-Object -ExpandProperty temp
Which is almost what I want, except it looks like the fields are arrays not strings. I am trying to have just one string for each element, ie (Name, Synopsis, Syntax, etc).
If I try to export that to csv, I get the lengths of the objects instead of the objects themselves:
Get-Help Get-Content | Select-Object -First 1 -Property #{
Name="temp"
Expression = { $_.Name,$_.Synopsis,$_.syntax,$_.Description }
} | Select-Object -ExpandProperty temp | export-csv -NoType -Path $env:HOME\test.txt
I suppose you want to export each property in a seperate column (your script currently only exports one). To do this, you have to select each property. For Syntax and Description I used the Out-String cmdlet to convert it to a string and removed all \r\n to get a valid CSV:
Get-Help Get-Content | Select-Object -First 1 |
Select-Object Name, Synopsis,
#{l='Syntax'; e={($_.Syntax | out-string) -replace "`r?`n"}},
#{l='Description '; e={($_.Description | out-string)-replace "`r?`n"}} |
Export-Csv -Path $env:HOME\test.txt -NoTypeInformation
I have below powershell code using which am saving results into csv file but I couldnt save them in csv file.
[Reflection.Assembly]::LoadWithPartialName("Microsoft.AnalysisServices")
$SSASServerName = "SSAS_ServerName"
$SSASDB = "TESTDB"
$SSASServer = New-Object Microsoft.AnalysisServices.Server
$SSASServer.Connect($SSASServerName)
$SSASDatabase = $SSASServer.Databases.Item($SSASDB)
$SSASDatabase.Roles | Select-Object Name, Members | Export-Csv C:\dev\psout\test.Csv
pause
This script extracts name of the role and members associated to that role. one role can have multiple members.
I tried above script, it exports role but in Members field, I see string "Microsoft.AnalysisServices.RoleMemberCollection" for all the roles.
If I do not export to csv, I can view the members in either ps window or text file.
what am i missing?
You can only export values that can be represented as a string to a csv-file. Members is a collection-object that may include multiple RoleMember-objects, so you need use a calculated property to access the Name-property inside each RoleMember. How to approach this depends on the desired output.
You can join the objects Name-property to a single string
$SSASDatabase.Roles |
Select-Object Name, #{n="Members";e={ ($_.Members | % { $_.Name }) -join '; '}} |
Export-Csv C:\dev\psout\test.Csv -NoTypeInformation
Role1,"User1; User2"
Role2,"User3; User4"
Or you could make one row in the csv-file "per row per member" which I usually prefer since it's easier to filter in Excel.
$SSASDatabase.Roles | ForEach-Object {
#Store role-reference so we can access it later inside the member-foreach
$r = $_
$r.Members | ForEach-Object {
#Store member-reference so it's available inside Selec-Object
$m = $_
$r | Select-Object Name, #{n="Member";e={ $m.Name }}
}
} | Export-Csv C:\dev\psout\test.Csv -NoTypeInformation
Role1,User1
Role1,User2
Role2,User3
...
I have a powershell script to find particular instances and then export them to CSV. Here's an example of the way the code works
$items = "Hello Tim", "Hola Bob", "Hello Susan"
$filter = $items | Select-String -Pattern "Hello"
$filter | Select-Object Line, Matches | Export-Csv "C:\log.csv"
Invoke-Item "C:\log.csv"
When I run the Select-Object in PS, it's nicely formatted info like this:
However, when I export to CSV, it exports the whole object and writes it as the following string: System.Text.RegularExpressions.Match[]
How can I get it to export just the first match or a listing of all matches into a single field when writing to CSV?
Here is one way using a PSObject:
$items = "Hello Tim", "Hola Bob", "Hello Susan"
$filter = $items | Select-String -Pattern "Hello"
$filter | % {New-Object PSObject -property #{
Line = $_.Line
Matches = $_.Matches.Groups[0].Value}
} | Export-Csv "C:\log.csv" -NoTypeInformation
Quickly note that Matches is an array which may create issues exporting to a csv.
Try joining the array into a string with a chosen delimiter. I used "::" in my example.
$filter | Select Line, #{Expression={$_.Matches -join "::"}; Label="Matches"} | Export-Csv "C:\log.csv"
I am storing the content of a JSON output into a variable and the by using:
$j1| Select - Object -Property #{Label = "id"; Expression = {$_.id} | Export-CSV -Path C:\Temp\j1.csv -Delimiter "|" -notype
I am exporting the values to csv file.
My issue is that inside the PowerShell, I can see e.g.
{1}
{2,3}
{4}
{5,6}
However, after exporting to csv, the comma delimiter of object is missing. And I would like to have in csv too.
Could you please help me with my problem?
Thanks in advance
You can't have arrays as properties when you export to CSV. You need to join the IDs to a string before exporting it. Ex:
Select-Object -Property #{Label="id";Expression={$_.id -join ','}