I created custom object that basically stores Date and few integers in it's keys:
$data = [Ordered]#{
"Date" = $currentdate.ToString('dd-MM-yyyy');
"Testers" = $totalTesterCount;
"StNoFeedback" = $tester_status.NoFeedback;
"StNotSolved" = $tester_status.NotSolved;
"StSolved" = $tester_status.Solved;
"StNoIssues" = $tester_status.NoIssues;
"OSNoFeedback" = $tester_os.NoFeedback;
"OSW7" = $tester_os.W7;
"OSW10" = $tester_os.W10;
"OfficeNoFeedback" = $tester_Office.NoFeedback;
"OfficeO10" = $tester_Office.O10;
"OfficeO13" = $tester_Office.O13;
"OfficeO16" = $tester_Office.O16;
}
I need to Output it to CSV file in a way that every value is written in new column.
I tried using $data | export-csv dump.csv
but my CSV looks like that:
#TYPE System.Collections.Specialized.OrderedDictionary
"Count","IsReadOnly","Keys","Values","IsFixedSize","SyncRoot","IsSynchronized"
"13","False","System.Collections.Specialized.OrderedDictionary+OrderedDictionaryKeyValueCollection","System.Collections.Specialized.OrderedDictionary+OrderedDictionaryKeyValueCollection","False","System.Object","False"
Not even close to what I want to achieve. How to get something closer to:
date,testers,stnofeedback....
04-03-2016,2031,1021....
I created the object because it was supposed to be easy to export it as csv. Maybe there is entirely different, better approach? Or is my object lacking something?
You didn't create an object, you created an ordered dictionary. A dictionary can't be exported to CSV-directly as it's a single object which holds multiple key-value-entries.
([ordered]#{}).GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True OrderedDictionary System.Object
To export a dictionary you need to use GetEnumerator() to get the objects one by one, which would result in this CSV:
$data = [Ordered]#{
"Date" = (get-date).ToString('dd-MM-yyyy')
"Testers" = "Hello world"
}
$data.GetEnumerator() | ConvertTo-Csv -NoTypeInformation
"Name","Key","Value"
"Date","Date","04-03-2016"
"Testers","Testers","Hello world"
If you want a single object, cast the hashtable of properties to a PSObject using [pscustomobject]#{}.
$data = [pscustomobject]#{
"Date" = (get-date).ToString('dd-MM-yyyy')
"Testers" = "Hello world"
}
$data | ConvertTo-Csv -NoTypeInformation
"Date","Testers"
"04-03-2016","Hello world"
Or if you're using PS 1.0 or 2.0:
$data = New-Object -TypeName psobject -Property #{
"Date" = (get-date).ToString('dd-MM-yyyy')
"Testers" = "Hello world"
}
$data | ConvertTo-Csv -NoTypeInformation
"Testers","Date"
"Hello world","04-03-2016"
Related
I would like to convert a log file to JSON format.
The content of the Log file is as below:
2021-07-13T14:32:00.197904 DDD client=10.4.35.4
2021-07-13T14:32:00.271923 BBB from=<josh.josh#test.com>
2021-07-13T14:32:01.350434 CCC from=<rob.roder#test.com>
2021-07-13T14:32:01.417904 DDD message-id=<1-2-3-a-a#A1>
2021-07-13T14:32:01.586494 DDD from=<Will.Smith#test.com>
2021-07-13T14:32:02.643101 DDD to=<Will.Smith#test.com>
2021-07-13T14:32:02.712803 AAA client=10.1.35.2
2021-07-13T14:32:03.832661 BBB client=2021:8bd::98e7:2e04:f94s
2021-07-13T14:32:03.920297 DDD status=sent
However the problem that occurs is that I need to match the IDs for each line to export to JSON that looks like:
{
"time": {
"start": "2021-07-13T14:32:01.417904",
"duration": "0:00:02.502393"
},
"sessionid": "DDD",
"client": "10.4.35.4",
"messageid": "<1-2-3-a-a#A1>",
"address": {
"from": "<Will.Smith#test.com>",
"to": "<Will.Smith#test.com>"
},
"status": "sent"
}
]
Next step is to import this data to analysis tool which only acceps JSON format. I've tried this with powershell and python, but got nowhere near the expected output.
The problems along the way:
How to link each row by session?
How to count 1st and last session duration?
How to link each session 3rd column results and how to convert them to json?
I would really appreciate any help, links, studies, etc.
You may do something similar to the following:
Get-Content a.log | Foreach-Object {
if ($_ -match '^(?<time>\S+)\s+(?<sessionid>\w+)\s+(?<key>[^=]+)=(?<data>.*)$') {
[pscustomobject]#{
time = $matches.time
sessionid = $matches.sessionid
key = $matches.key
data = $matches.data
}
}
} | Group sessionid | Foreach-Object {
$jsonTemplate = [pscustomobject]#{
time = [pscustomobject]#{ start = ''; duration = '' }
sessionid = ''
client = ''
messageid = ''
address = [pscustomobject]#{from = ''; to = ''}
status = ''
}
$start = ($_.Group | where key -eq 'message-id').time
$end = ($_.Group | where key -eq 'status').time -as [datetime]
$jsonTemplate.time.start = $start
$jsonTemplate.time.duration = ($end - ($start -as [datetime])).ToString()
$jsonTemplate.sessionid = $_.Name
$jsonTemplate.client = ($_.Group | where key -eq 'client').data
$jsonTemplate.messageid = ($_.Group | where key -eq 'message-id').data
$jsonTemplate.address.from = ($_.Group | where key -eq 'from').data
$jsonTemplate.address.to = ($_.Group | where key -eq 'to').data
$jsonTemplate.status = ($_.Group | where key -eq 'status').data
[regex]::Unescape(($jsonTemplate | convertTo-Json))
}
The general steps that are happening are the following:
Parse the log file to separate data elements
Group by sessionid to easily identify all event entries belonging to the session id
Create a custom object that contains the schema that'll easily convert to the desired JSON format.
The regex unescape is to remove the unicode escape codes for the < and > characters.
The $matches automatic variable updates when the -match operation returns $true. Since we are using named capture groups in the regex expression, capture groups are accessible as keys in the $matches hashtable.
Caveats:
Assumes sessionid only has one session per id.
Missing session data shows up as null in JSON format.
Alternative solutions may use ConvertFrom-String when reading the file. It is just simpler for me personally to do regex matching instead.
A solution based on a switch statement, which enables fast line-by-line processing with its -File parameter:
A nested (ordered) hashtable is used to compile session-specific information across lines.
The -split operator is used to split each line into fields, and to split the last field into property name and value.
Note:
The calculation of the session duration assumes that the first line for a given session ID marks the start of the session, and a line with a status= value the end.
$sessions = [ordered] #{}
switch -File file.log { # process file line by line
default {
$timestamp, $sessionId, $property = -split $_ # split the line into fields
$name, $value = $property -split '=', 2 # split the property into name an value
if ($session = $sessions.$sessionId) { # entry for session ID already exists
$session.$name = $value
# end of session? calculate the duration
if ($name -eq 'status') { $session.time.duration = ([datetime] $timestamp - [datetime] $session.time.start).ToString() }
}
else { # create new entry for this session ID
$sessions.$sessionId = [ordered] #{
$name = $value
time = [ordered] #{
start = $timestamp
duration = $null
}
}
}
}
}
# Convert the hashtable to JSON
$sessions | ConvertTo-Json
I have an appsettings.json file that I would like to transform with a PowerShell script in a VSTS release pipeline PowerShell task. (BTW I'm deploying a netstandard 2 Api to IIS). The JSON is structured like the following:
{
"Foo": {
"BaseUrl": "http://foo.url.com",
"UrlKey": "12345"
},
"Bar": {
"BaseUrl": "http://bar.url.com"
},
"Blee": {
"BaseUrl": "http://blee.url.com"
}
}
I want to replace BaseUrl and, if it exists, the UrlKey values in each section which are Foo, Bar and Blee. (Foo:BaseUrl, Foo:UrlKey, Bar:BaseUrl, etc.)
I'm using the following JSON structure to hold the new values:
{
"##{FooUrl}":"$(FooUrl)",
"##{FooUrlKey}":"$(FooUrlKey)",
"##{BarUrl}":"$(BarUrl)",
"##{BleeUrl}":"$(BleeUrl)"
}
So far I have the following script:
# Get file path
$filePath = "C:\mywebsite\appsettings.json"
# Parse JSON object from string
$jsonString = "$(MyReplacementVariablesJson)"
$jsonObject = ConvertFrom-Json $jsonString
# Convert JSON replacement variables object to HashTable
$hashTable = #{}
foreach ($property in $jsonObject.PSObject.Properties) {
$hashTable[$property.Name] = $property.Value
}
# Here's where I need some help
# Perform variable replacements
foreach ($key in $hashTable.Keys) {
$sourceFile = Get-Content $filePath
$sourceFile -replace $key, $hashTable[$key] | Set-Content $filePath
Write-Host 'Replaced key' $key 'with value' $hashTable[$key] 'in' $filePath
}
Why are you defining your replacement values as a JSON string? That's just going to make your life more miserable. If you're defining the values in your script anyway just define them as hashtables right away:
$newUrls = #{
'Foo' = 'http://newfoo.example.com'
'Bar' = 'http://newbaz.example.com'
'Blee' = 'http://newblee.example.com'
}
$newKeys = #{
'Foo' = '67890'
}
Even if you wanted to read them from a file you could make that file a PowerShell script containing those hashtables and dot-source it. Or at least define the values as lists of key=value lines in text files, which can easily be turned into hashtables:
$newUrls = Get-Content 'new_urls.txt' | Out-String | ConvertFrom-StringData
$newKeys = Get-Content 'new_keys.txt' | Out-String | ConvertFrom-StringData
Then iterate over the top-level properties of your input JSON data and replace the nested properties with the new values:
$json = Get-Content $filePath | Out-String | ConvertFrom-Json
foreach ($name in $json.PSObject.Properties) {
$json.$name.BaseUrl = $newUrls[$name]
if ($newKeys.ContainsKey($name)) {
$json.$name.UrlKey = $newKeys[$name]
}
}
$json | ConvertTo-Json | Set-Content $filePath
Note that if your actual JSON data has more than 2 levels of hierarchy you'll need to tell ConvertTo-Json via the parameter -Depth how many levels it's supposed to convert.
Side note: piping the Get-Content output through Out-String is required because ConvertFrom-Json expects JSON input as a single string, and using Out-String makes the code work with all PowerShell versions. If you have PowerShell v3 or newer you can simplify the code a little by replacing Get-Content | Out-String with Get-Content -Raw.
Thank you, Ansgar for your detailed answer, which helped me a great deal. Ultimately, after having no luck iterating over the top-level properties of my input JSON data, I settled on the following code:
$json = (Get-Content -Path $filePath) | ConvertFrom-Json
$json.Foo.BaseUrl = $newUrls["Foo"]
$json.Bar.BaseUrl = $newUrls["Bar"]
$json.Blee.BaseUrl = $newUrls["Blee"]
$json.Foo.Key = $newKeys["Foo"]
$json | ConvertTo-Json | Set-Content $filePath
I hope this can help someone else.
To update values of keys at varying depth in the json/config file, you can pass in the key name using "." between the levels, e.g. AppSettings.Setting.Third to represent:
{
AppSettings = {
Setting = {
Third = "value I want to update"
}
}
}
To set the value for multiple settings, you can do something like this:
$file = "c:\temp\appSettings.json"
# define keys and values in hash table
$settings = #{
"AppSettings.SettingOne" = "1st value"
"AppSettings.SettingTwo" = "2nd value"
"AppSettings.SettingThree" = "3rd value"
"AppSettings.SettingThree.A" = "A under 3rd"
"AppSettings.SettingThree.B" = "B under 3rd"
"AppSettings.SettingThree.B.X" = "Z under B under 3rd"
"AppSettings.SettingThree.B.Y" = "Y under B under 3rd"
}
# read config file
$data = Get-Content $file -Raw | ConvertFrom-Json
# loop through settings
$settings.GetEnumerator() | ForEach-Object {
$key = $_.Key
$value = $_.Value
$command = "`$data.$key = $value"
Write-Verbose $command
# update value of object property
Invoke-Expression -Command $command
}
$data | ConvertTo-Json -Depth 10 | Out-File $file -Encoding "UTF8"
I have a powershell script in which I do the following
$somePSObjectHashtables = New-Object Hashtable[] $somePSObject.Length;
$somePSObjects = Import-CSV $csvPath
0..($somePSObject.Length - 1) | ForEach-Object {
$i = $_;
$somePSObjectHashtables[$i] = #{};
$somePSObject[$_].PSObject.Properties | ForEach-Object {
$somePSObjectHashtables[$i][$_.Name] = $_.Value;
}
}
I need to do this because I want to make several distinct copies of the data in the CSV to perform several distinct manipulations. In a sense I'm performing an "INNER JOIN" on the resulting array of PSObject. I can easily iterate through $somePSObjectHashtables with a ForEach-Object and call Hashtable.Clone() on each member of the array. I can then use New-Object PSObject -Property $someHashTable[$i] to get a deep copy of the PSObject.
My question is, is there some easier way of making the deep copy, without an intermediary Hashtable?
Note that here is a shorter, maybe a bit cleaner version of this (that I quite enjoy):
$data = Import-Csv .\test.csv
$serialData = [System.Management.Automation.PSSerializer]::Serialize($data)
$data2 = [System.Management.Automation.PSSerializer]::Deserialize($serialData)
Note:
However, weirdly, it does not keep the ordering of ordered hashtables.
$data = [ordered] #{
1 = 1
2 = 2
}
$serialData = [System.Management.Automation.PSSerializer]::Serialize($data)
$data2 = [System.Management.Automation.PSSerializer]::Deserialize($serialData)
$data2
Will output:
Name Value
---- -----
2 2
1 1
While with other types it works just fine:
$data = [PsCustomObject] #{
1 = 1
2 = 2
}
$data = #(1, 2, 3)
For getting really deep copies we can use binary serialization (assuming that all data are serializable; this is definitely the case for data that come from CSV):
# Get original data
$data = Import-Csv ...
# Serialize and Deserialize data using BinaryFormatter
$ms = New-Object System.IO.MemoryStream
$bf = New-Object System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
$bf.Serialize($ms, $data)
$ms.Position = 0
$data2 = $bf.Deserialize($ms)
$ms.Close()
# Use deep copied data
$data2
Here's an even shorter one that I use as a function:
using namespace System.Management.Automation
function Clone-Object ($InputObject) {
<#
.SYNOPSIS
Use the serializer to create an independent copy of an object, useful when using an object as a template
#>
[psserializer]::Deserialize(
[psserializer]::Serialize(
$InputObject
)
)
}
Suppose I read in a CSV file from PowerShell:
$data = Import-Csv "myfilename.csv"
CSV files (in general) can contain strings and numbers, but PowerShell stores them in memory as strings:
PS D:\> $data[0].Col3.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
After importing, it would be useful to be able to convert the types from string. If there are only one or two columns then I can convert them using a calculated property as follows:
$data = Import-Csv "myfilename.csv" |
select -Property #{n='Col2';e={[int]$_.Col2}},
#{n='Col3';e={[double]$_.Col3}}
But suppose I don't know in advance the column names and intended types. Instead I have an arbitrary "schema" telling me which columns should be which type, for example:
$Schema = #{Col1=[string];Col2=[int];Col3=[double]}
How can I convert the output from Import-CSV to the types as determined by the schema? (And preferably in an efficient/elegant way)
Sample CSV file
"Col1","Col2","Col3"
"a",2,4.3
"b",5,7.9
You can do this with a -as cast:
$data = Import-Csv "myfilename.csv" |
select -Property #{n='Col2';e={$_.Col2 -as $Schema.Col2}},
#{n='Col3';e=$_.Col3 -as $Schema.Col3}}
For an arbitrary number of columns you can extend the approach outlined in this answer to a similar question:
$data = Import-Csv "myfilename.csv" | Foreach-Object {
foreach ($property in $_.PSObject.Properties) {
$property.Value = $property.Value -as $Schema[$property.Name]
}
$_ # return the modified object
}
I expanded upon Martin Brandl's great answer here in two ways:
First, it can handle more complex cases. Instead of having the schema be a hash table of data types, I generalized it to be a hash table of conversion functions. This allows you to do non-trivial data-type conversions as well as per-column pre-processing/clean-up.
I also flipped the for-each logic so that it iterates through the schema keys instead of the object's properties. That way, your schema doesn't need to contain every field, which is helpful if you have a CSV with many string fields that can be left alone and just a few fields that need data type conversion.
In the example below:
The text column is purposefully left out of the schema to demonstrate that that's ok.
The memory column is converted from bytes to kilobytes.
The third column is converted from sloppy strings to booleans.
Example
$testData = #(
[PSCustomObject]#{Text = 'A'; MemoryWithConversion = 10*1024; BooleanText="yes"},
[PSCustomObject]#{Text = 'B'; MemoryWithConversion = 20*1024; BooleanText="no"},
[PSCustomObject]#{Text = 'C'; MemoryWithConversion = 30*1024; BooleanText=""}
)
$testData | Export-Csv 'test.csv'
$schema = #{
MemoryWithConversion = {Param($value) $value / 1kB -as [int]}
BooleanText = {Param($value) $value -in 'true', 't', 'yes', 'y' -as [boolean]}
}
Import-Csv 'test.csv' | Foreach-Object {
foreach ($key in $schema.Keys) {
$property = $_.PSObject.Properties[$key]
if ($property -ne $null) {
$property.Value = & $schema[$property.Name] $property.Value
}
}
$_
}
Result
Text MemoryWithConversion BooleanText
---- -------------------- -----------
A 10 True
B 20 False
C 30 False
I want to be able to Import-Csv into a PowerShell table so I can edit each section with PowerShell script, e.g. $table.row[0].name = 100. Import-Csv doesn't give that "table" it makes with CSV file.
$tabName = "TableA"
$table = New-Object system.Data.DataTable "$tabName"
$col1 = New-Object system.Data.DataColumn ColumnName1,([string])
$col2 = New-Object system.Data.DataColumn ColumnName2,([int])
$table.columns.add($col1)
$table.columns.add($col2)
$row = $table.NewRow()
$row.ColumnName1 = "A"
$row.ColumnName2 = "1"
$table.Rows.Add($row)
$row = $table.NewRow()
$row.ColumnName1 = "B"
$row.ColumnName2 = "2"
$table.Rows.Add($row)
$table.PrimaryKey = $table.Columns[0]
$table | Export-Csv C:\test.csv
I want a way to $table | Import-Csv C:\test.csv and have the ability to $table.row[0].columnname1 = "C" and it changes "A" to "C". Then I can re-export it after making changes.
You can export a DataTable object to CSV simply by piping the table into the Export-Csv cmdlet:
$table | Export-Csv C:\table.csv -NoType
However, by doing that you lose all type information of the table columns (the Export-Csv cmdlet can only save information about the type of the objects that represent the rows, not about the type of their properties).
A better way to save and restore a DataTable object is to save the table as XML:
$writer = New-Object IO.StreamWriter 'C:\path\to\data.xml'
$table.WriteXml($writer, [Data.XmlWriteMode]::WriteSchema)
$writer.Close()
$writer.Dispose()
and restore the XML into a DataSet:
$ds = New-Object Data.DataSet
$ds.ReadXml('C:\path\to\data.xml', [Data.XmlReadMode]::ReadSchema)
$table = $ds.Tables[0]
Make sure to export and import the schema along with the data, because that's where the type information is stored.
The result of import-csv, as you found, is not a .NET DataTable object. There is this function to convert to a DataTable called "Out-DataTable" https://gallery.technet.microsoft.com/scriptcenter/4208a159-a52e-4b99-83d4-8048468d29dd . I don't know that it can go both directions, though.