Powershell: How to create a JSON in the GeoJSON structure - json

I'm having problems creating a GeoJSON file. I'm looping through video files extracting info such as datetaken, name, coordinates, etc. I want to write this data to a JSON file so I can plot them on googlemaps. But I'm stuck. I can't get the lat/lng in the correct json format.
The GeoJSON structure is as follows:
"features": [{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [125.6, 10.1]
},
"properties": {
"name": "Lost Islands"
}
]}
My code, no matter what I try, generates the following or something similar.
"features": [{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": "System.Collections.Hashtable"
},
"properties": {
"name": "Lost Island"
}
]}
My code (simplified for here):
cls
$dirName = "C:\temp"
$filter = "*.mp4"
$jsonBase = #{}
$features = New-Object System.Collections.ArrayList
$coordinatesList = New-Object System.Collections.ArrayList
$coordinates = New-Object System.Collections.ArrayList
$geometry = New-Object System.Collections.ArrayList
Get-ChildItem $dirName -include $filter -recurse -File | foreach {
$lat = 10.12345
$lng = 20.12345
$vidDate = "2022-09-11T12:00:00.000000Z"
$coordinatesList = New-Object System.Collections.ArrayList
$coordinatesList.Add(#{$lat=$lng;})
$coordinates = New-Object System.Collections.ArrayList
$coordinates.Add(#{"coordinates"=$coordinatesList})
$geometry = #{}
$geometry.Add("coordinates",$coordinates)
$geometry.Add("type","Point")
$features.Add(#{"geometry"=$geometry;
"type" = "Feature";
"properties" = #{"fileName"=($_).Name;"pathToFile"=($_).FullName;"takenDate"=$vidDate;}
})
}
$jsonBase.Add("features",$features)
$jsonstring = $jsonBase | ConvertTo-Json -Depth 3
Write-Host $jsonstring
Any thoughts on where I'm going wrong?

Thanks to mclayton for spotting the depth mistake.
I still had to make a small change to get it in the correct format. Here's the final code:
cls
$dirName = "C:\temp"
$filter = "*.mp4"
$jsonBase = #{}
$features = New-Object System.Collections.ArrayList
$geometry = New-Object System.Collections.ArrayList
Get-ChildItem $dirName -include $filter -recurse -File | foreach {
$lat = 10.12345
$lng = 20.12345
$vidDate = "2022-09-11T12:00:00.000000Z"
$geometry = #{}
$geometry.Add("coordinates",($lat,$lng))
$geometry.Add("type","Point")
$features.Add(#{"geometry"=$geometry;
"type" = "Feature";
"properties" = #{"fileName"=($_).Name;"pathToFile"=($_).FullName;"takenDate"=$vidDate;}
})
}
$jsonBase.Add("features",$features)
$jsonstring = $jsonBase | ConvertTo-Json -Depth 10
Write-Host $jsonstring

The issue is in this line:
$coordinatesList.Add(#{ $lat = $lng })
$lat will become a double type Key on your hashtable and ConvertTo-Json only allows string type keys, probably you got this error but didn't notice it:
$lat = 10.12345
$lng = 20.12345
#{ $lat = $lng } | ConvertTo-Json
Errors with
The type 'System.Collections.Hashtable' is not supported for serialization or deserialization of a dictionary. Keys must be strings.
But also, looking at your expected Json, you most likely wanted coordinates to be an array instead of a key / value pair. Here is a streamlined version of your code:
#{
features = #(Get-ChildItem $dirName -Filter $filter -Recurse -File | ForEach-Object {
$lat = 10.12345
$lng = 20.12345
$vidDate = "2022-09-11T12:00:00.000000Z"
[ordered]#{
type = 'Feature'
geometry = [ordered]#{
type = 'Point'
coordinates = $lat, $lng # Should be Array Here
}
properties = [ordered]#{
fileName = $_.Name
pathToFile = $_.FullName
takenDate = $vidDate
}
}
}
)} | ConvertTo-Json -Depth 99

Related

How to insert bulk document into elastic using PowerShell?

I am trying to read the data from CSV file which has 2200000 records using PowerShell and storing each record in JSON file as NDJSON format.
Total document in JSON file is 2200000 (200 MB)
Sample JSON Data
{"index":{}}
{"ip-address":"1.5.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8325021+05:30"}
{"index":{}}
{"ip-address":"243.11.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
{"index":{}}
{"ip-address":"253.11.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
{"index":{}}
{"ip-address":"39.24.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
{"index":{}}
{"ip-address":"163.24.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
Code
function Get-IPDataPath
{
$dbFilePath = Get-ChildItem -Path $rootDir -Filter "IP2*.CSV" | ForEach-Object FullName | Select-Object -First 1
Write-Host "file path - $dbFilePath"
$dbFilePath # implicit output
}
function Convert-NumberToIP
{
param(
[Parameter(Mandatory=$true)][string]$number
)
[Int64] $numberInt = 0
if( [Int64]::TryParse( $number, [ref] $numberInt ) ) {
if( ($numberInt -ge 0) -and ($numberInt -le 0xFFFFFFFFl) ) {
# Convert to IP address like '192.168.23.42'
([IPAddress] $numberInt).ToString()
}
}
# In case TryParse() returns $false or the number is out of range for an IPv4 address,
# the output of this function will be empty, which converts to $false in a boolean context.
}
function Insert-Document
{
param(
[Parameter(Mandatory=$true)][string]$indexName,
[Parameter(Mandatory=$true)][object]$filePath
)
$url = "https://esdev2:9200/naveen/_doc/_bulk"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($cred))
$headers = #{
"Authorization" = "Basic $encodedCreds"
"Transfer-Encoding" = "chunked"
"Content-Type" = "application/x-ndjson"
}
$content = Get-Content -Path $filepath
$response = Invoke-WebRequest -Uri $url -Method POST -Body $content -Headers $Headers -ContentType 'application/x-ndjson'
if($response.StatusCode -eq 201)
{
Write-Host "Documents added successfully"
}
else
{
Write-Host $response
Write-Host "$($response.content)"
throw "Trouble adding document: $_"
}
}
$dbFilePath = Get-IPDataPath
$outputFile = Join-Path -Path $rootDir -ChildPath "output.json"
Write-Host "Converting CSV file $dbFilePath to $outputFile"
$object = [PSCustomObject]#{
'ip-address' = ''
'is-vpn' = 'true'
'#timestamp' = ''
}
# Enclose foreach loop in a script block to be able to pipe its output to Set-Content
$count = 0
& {
foreach( $item in [Linq.Enumerable]::Skip( [IO.File]::ReadLines( $dbFilePath ), 1 ) )
{
$row = $item -split ','
$ipNumber = $row[0].trim('"')
if( $ip = Convert-NumberToIP -number $ipNumber )
{
$index = [PSCustomObject]#{"index" = [PSCustomObject]#{}} | ConvertTo-Json -Depth 10 -Compress
$object.'ip-address' = $ip
$object.'#timestamp' = (Get-Date).ToString('o')
#$index | ConvertTo-Json -Depth 100 -Compress
$document = $object | ConvertTo-Json -Depth 100 -Compress
if($count -lt 25)
{
Add-Content -Path $outputFile $index
Add-Content -Path $outputFile $document
$count++
}
else
{
$count = 0
BulkInsert-Document -indexName $indexName -filePath $outputFile
Clear-Content -Path $outputFile
}
}
}
} | Set-Content -Path $outputFile
Write-Host "Inserting document in elastic"
Insert-Document -indexName $indexName -filePath $outputFile
But I am getting Error: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk
request must be terminated by a newline
[\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be
terminated by a newline [\n]"},"status":400}
Though I have new line at the end of JSON file, Could you please help how to insert all the document.

Powershell Convert to JSON

I want to convert the data in the following script to JSON, can anyone let me know how to include [] in the JSON using powershell.
My script:
$URL = " http://localhost:4200/api/testinglabproject/test-run"
$projectShortName = "NES"
$testPlanName = "TestManager"
$deviceName = "testt"
$nodeManagerResourceName = "nm-4c2f1575-00fd-4363-b88a-2fb3cf587223"
$hostname = "DSTW8Y2.bla.is.abc.com"
$resourceName = "hs-515a8129-1aa0-4a14-847b-210022fe9cd7"
$hardwareSolutionType = "testt"
$emailId = "abc#gmail.com"
$data = #{
projectShortName=$projectShortName
testPlanName=$testPlanName
hardwareSolution=#{
hardwareSolutionItemList =
#{deviceName = $deviceName
nodeManagerResourceName =$nodeManagerResourceName
hostname = $hostname}
hardwareSolutionMetadataList = "[]"
resourceName = $resourceName
hardwareSolutionType = $hardwareSolutionType
}
emailId = $emailId
}
$jsondata = $data | ConvertTo-Json -Depth 3
Write-Host $jsondata
The resulting JSON should be like this:
{
"projectShortName": <projectShortName>,
"testPlanName": <planName>,
"hardwareSolution": {
"hardwareSolutionItemList": [{"deviceName": <deviceName>,
"nodeManagerResourceName": <NodeManagerResourceName>,
"hostname": <hostName>}],
"hardwareSolutionMetadataList": [],
"resourceName": <HardwareSolutionResourceName>,
"hardwareSolutionType": <hwstName>
},
"emailId": <emailID>
}
I stripped all those superfluous variables and replaced the string "[]" with an actual PowerShell array #() for the hardwareSolutionMetadataList key:
$data = #{
testPlanName = "TestManager"
projectShortName = "NES"
hardwareSolution = #{
hardwareSolutionItemList = #{
deviceName = "testt"
nodeManagerResourceName = "nm-4c2f1575-00fd-4363-b88a-2fb3cf587223"
hostname = "DSTW8Y2.bla.is.abc.com"
}
hardwareSolutionMetadataList = #()
resourceName = "hs-515a8129-1aa0-4a14-847b-210022fe9cd7"
hardwareSolutionType = "testt"
}
emailId = "abc#gmail.com"
}
$jsondata = $data | ConvertTo-Json -Depth 3
Write-Host $jsondata
PowerShell's idea of JSON formatting is a bit odd, but structurally it's OK.
{
"hardwareSolution": {
"hardwareSolutionItemList": {
"deviceName": "testt",
"hostname": "DSTW8Y2.bla.is.abc.com",
"nodeManagerResourceName": "nm-4c2f1575-00fd-4363-b88a-2fb3cf587223"
},
"resourceName": "hs-515a8129-1aa0-4a14-847b-210022fe9cd7",
"hardwareSolutionMetadataList": [
],
"hardwareSolutionType": "testt"
},
"testPlanName": "TestManager",
"emailId": "abc#gmail.com",
"projectShortName": "NES"
}
You can use the -Compress parameter to remove the whitespace for uploading the JSON somewhere.
{"hardwareSolution":{"hardwareSolutionItemList":{"deviceName":"testt","hostname":"DSTW8Y2.bla.is.abc.com","nodeManagerResourceName":"nm-4c2f1575-00fd-4363-b88a-2fb3cf587223"},"resourceName":"hs-515a8129-1aa0-4a14-847b-210022fe9cd7","hardwareSolutionMetadataList":[],"hardwareSolutionType":"testt"},"testPlanName":"TestManager","emailId":"abc#gmail.com","projectShortName":"NES"}

Replace parameters in JSON with given Values from .csv

I have a JSON-File where I want to replace several values with a placeholder.
So i made a .csv with the parameters to replace. line[0] (if existing) is the path in the file, line[1] the element, line[2] the placeholder.
journeys.legs.origin.properties.downloads;url;placeholder;
;url;placeholder;
download;url;placeholder;
;psFileName;placeholder;
;serverTime;placeholder;
;calcTime;placeholder;
now I defined the following functions, to get the file, read the csv and replace the stuff.
$storage = "D:\Service\test.json"
$parameters2replace = "D:\Service\parameters2replace.csv"
function Get-JSONProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
function setParameter(){
foreach ($parameter in Get-Content $parameters2replace){
$line=$parameter.split(";")
$path = $line[0]
$elementName = $line[1]
$newValue = $line[2]
replaceElement $path $elementName $newValue
}
}
function replaceElement($path, $elementName, $newValue){
ForEach($JSONPath in Get-JSONProperty $JSON $path){
if (!$line[0]){
if($JSON.$elementName){
$JSON.$elementName = $newValue
}
}
else{
if($JSON.$path.$elementName){
echo $JSON.$path.$elementName
$JSON.$path.$elementName = $newValue
}
}
}
$JSON | ConvertTo-Json -depth 32| set-content $storage
}
$JSON = Get-Content $storage -raw | ConvertFrom-Json
setParameter
My problem now is, that the following if-argument won't work with the $path variable. If i put it in hardcoded it works just fine.
if($JSON.$path.$elementName)
I hope i could make everything clear, this was my first post.
$JSON.$path makes PowerShell look for a property with the literal name 'journeys.legs.origin.properties.downloads'. You need to split the string into it's individual parts and iterate (or recurse) through the path:
function Get-JSONPropertyValue {
param(
$JSON,
[string]$Path,
[string]$Name
)
$object = $JSON
foreach($propName in $Path.Split('.')){
$object = $object.$propName
}
return $object.$Name
}
function Set-JSONPropertyValue
{
param(
$JSON,
[string]$Path,
[string]$Name,
$Value
)
$object = $JSON
foreach($propName in $Path.Split('.')){
$object = $object.$propName
}
$object.$Name = $Value
}
Now you can do:
if($value = Get-JSONPropertyValue $JSON -Path $path -Name $elementName){
echo "Old value: $value"
Set-JSONPropertyValue $JSON -Path $path -Name $elementName -Value $newValue
}

Powershell directory structured into json format

Hello I am new to powershell.
I am trying to create a directory structured into json format using powershell. similar to these picture below
Example json format of directory structure
I have researched some ways into doing this and found this, is their a similar or better way into naming directory and files into json format.
function Add-Tabstops{
param($Count)
$tabs = ""
for($i=0; $i -lt $Count; $i++){$tabs += " "}
return $tabs
}
function Output-JsonChildren{
param($Path, $Level = 1)
return $(Get-ChildItem -Path $Path | Where-Object{$_} | ForEach-Object{
(Add-Tabstops $Level) +
"{`n" +
(Add-Tabstops ($Level+1)) +
"`"name`"`: `"$($_.Name)`"," +
"`n" +
(Add-Tabstops ($Level+1)) +
"`"children`": ["+
$(if($_.psiscontainer){"`n" + (Output-JsonChildren -Path $_.FullName -Level ($Level+2))+ "`n" + (Add-Tabstops ($Level+1))}) +
"]`n" +
(Add-Tabstops ($Level)) +
"}"
}) -join ",`n"
}
$JSON = Output-JsonChildren -Path "C:\Users\Glen\Desktop\democontainer" | Out-File "C:\Users\Glen\Desktop\democontainer\test.json"
"["
$JSON
"]"
Something like this:
function build-json {
param (
[string]$fullPath
)
$hashtable = #{}
$directories = get-childitem $fullPath -Directory
try{
$files = (get-childitem $fullPath -File)
if($null -eq $files) {
$hashtable.files = #()
}
else {
$hashtable.files = [array]$files.Name
}
}
catch {
echo 'xxx'
}
$hashtable.directories = #{}
foreach($directory in $directories) {
$element = Split-Path $directory.fullname -Leaf
$hashtable.directories.$element = build-json -fullPath $directory.FullName -hashtable $hashtable
}
return $hashtable
}
$sourcefolder = 'E:\xxx\Software\yyy\python392'
$result3 = build-json -fullPath $sourcefolder -hashtable #{}
Yes, it's possible.
You will need to write a little recursive function which walks through the dictionaries and collects the information for you.
You can use a hashtable (#{}) or a PowerShell object, but a hashtable is preferable.
Get-ChildItem and ConvertTo-Json are your friends

powershell convert json to hashtable recursively [duplicate]

What is the easiest way to convert a PSCustomObject to a Hashtable? It displays just like one with the splat operator, curly braces and what appear to be key value pairs. When I try to cast it to [Hashtable] it doesn't work. I also tried .toString() and the assigned variable says its a string but displays nothing - any ideas?
Shouldn't be too hard. Something like this should do the trick:
# Create a PSCustomObject (ironically using a hashtable)
$ht1 = #{ A = 'a'; B = 'b'; DateTime = Get-Date }
$theObject = new-object psobject -Property $ht1
# Convert the PSCustomObject back to a hashtable
$ht2 = #{}
$theObject.psobject.properties | Foreach { $ht2[$_.Name] = $_.Value }
Keith already gave you the answer, this is just another way of doing the same with a one-liner:
$psobject.psobject.properties | foreach -begin {$h=#{}} -process {$h."$($_.Name)" = $_.Value} -end {$h}
Here's a version that works with nested hashtables / arrays as well (which is useful if you're trying to do this with DSC ConfigurationData):
function ConvertPSObjectToHashtable
{
param (
[Parameter(ValueFromPipeline)]
$InputObject
)
process
{
if ($null -eq $InputObject) { return $null }
if ($InputObject -is [System.Collections.IEnumerable] -and $InputObject -isnot [string])
{
$collection = #(
foreach ($object in $InputObject) { ConvertPSObjectToHashtable $object }
)
Write-Output -NoEnumerate $collection
}
elseif ($InputObject -is [psobject])
{
$hash = #{}
foreach ($property in $InputObject.PSObject.Properties)
{
$hash[$property.Name] = ConvertPSObjectToHashtable $property.Value
}
$hash
}
else
{
$InputObject
}
}
}
My extremely lazy approach, enabled by a new feature in PowerShell 6:
$myhashtable = $mypscustomobject | ConvertTo-Json | ConvertFrom-Json -AsHashTable
This works for PSCustomObjects created by ConvertFrom_Json.
Function ConvertConvertFrom-JsonPSCustomObjectToHash($obj)
{
$hash = #{}
$obj | Get-Member -MemberType Properties | SELECT -exp "Name" | % {
$hash[$_] = ($obj | SELECT -exp $_)
}
$hash
}
Disclaimer: I barely understand PowerShell so this is probably not as clean as it could be. But it works (for one level only).
My code:
function PSCustomObjectConvertToHashtable() {
param(
[Parameter(ValueFromPipeline)]
$object
)
if ( $object -eq $null ) { return $null }
if ( $object -is [psobject] ) {
$result = #{}
$items = $object | Get-Member -MemberType NoteProperty
foreach( $item in $items ) {
$key = $item.Name
$value = PSCustomObjectConvertToHashtable -object $object.$key
$result.Add($key, $value)
}
return $result
} elseif ($object -is [array]) {
$result = [object[]]::new($object.Count)
for ($i = 0; $i -lt $object.Count; $i++) {
$result[$i] = (PSCustomObjectConvertToHashtable -object $object[$i])
}
return ,$result
} else {
return $object
}
}
For simple [PSCustomObject] to [Hashtable] conversion Keith's Answer works best.
However if you need more options you can use
function ConvertTo-Hashtable {
<#
.Synopsis
Converts an object to a hashtable
.DESCRIPTION
PowerShell v4 seems to have trouble casting some objects to Hashtable.
This function is a workaround to convert PS Objects to [Hashtable]
.LINK
https://github.com/alainQtec/.files/blob/main/src/scripts/Converters/ConvertTo-Hashtable.ps1
.NOTES
Base ref: https://community.idera.com/database-tools/powershell/powertips/b/tips/posts/turning-objects-into-hash-tables-2
#>
PARAM(
# The object to convert to a hashtable
[Parameter(ValueFromPipeline = $true, Mandatory = $true)]
$InputObject,
# Forces the values to be strings and converts them by running them through Out-String
[switch]$AsString,
# If set, empty properties are Included
[switch]$AllowNulls,
# Make each hashtable to have it's own set of properties, otherwise,
# (default) each InputObject is normalized to the properties on the first object in the pipeline
[switch]$DontNormalize
)
BEGIN {
$headers = #()
}
PROCESS {
if (!$headers -or $DontNormalize) {
$headers = $InputObject | Get-Member -type Properties | Select-Object -expand name
}
$OutputHash = #{}
if ($AsString) {
foreach ($col in $headers) {
if ($AllowNulls -or ($InputObject.$col -is [bool] -or ($InputObject.$col))) {
$OutputHash.$col = $InputObject.$col | Out-String -Width 9999 | ForEach-Object { $_.Trim() }
}
}
} else {
foreach ($col in $headers) {
if ($AllowNulls -or ($InputObject.$col -is [bool] -or ($InputObject.$col))) {
$OutputHash.$col = $InputObject.$col
}
}
}
}
END {
return $OutputHash
}
}
Maybe this is overkill but I hope it Helps
Today, the "easiest way" to convert PSCustomObject to Hashtable would be so:
$custom_obj | ConvertTo-HashtableFromPsCustomObject
OR
[hashtable]$custom_obj
Conversely, you can convert a Hashtable to PSCustomObject using:
[PSCustomObject]$hash_table
Only snag is, these nifty options may not be available in older versions of PS