Powershell directory structured into json format - json

Hello I am new to powershell.
I am trying to create a directory structured into json format using powershell. similar to these picture below
Example json format of directory structure
I have researched some ways into doing this and found this, is their a similar or better way into naming directory and files into json format.
function Add-Tabstops{
param($Count)
$tabs = ""
for($i=0; $i -lt $Count; $i++){$tabs += " "}
return $tabs
}
function Output-JsonChildren{
param($Path, $Level = 1)
return $(Get-ChildItem -Path $Path | Where-Object{$_} | ForEach-Object{
(Add-Tabstops $Level) +
"{`n" +
(Add-Tabstops ($Level+1)) +
"`"name`"`: `"$($_.Name)`"," +
"`n" +
(Add-Tabstops ($Level+1)) +
"`"children`": ["+
$(if($_.psiscontainer){"`n" + (Output-JsonChildren -Path $_.FullName -Level ($Level+2))+ "`n" + (Add-Tabstops ($Level+1))}) +
"]`n" +
(Add-Tabstops ($Level)) +
"}"
}) -join ",`n"
}
$JSON = Output-JsonChildren -Path "C:\Users\Glen\Desktop\democontainer" | Out-File "C:\Users\Glen\Desktop\democontainer\test.json"
"["
$JSON
"]"

Something like this:
function build-json {
param (
[string]$fullPath
)
$hashtable = #{}
$directories = get-childitem $fullPath -Directory
try{
$files = (get-childitem $fullPath -File)
if($null -eq $files) {
$hashtable.files = #()
}
else {
$hashtable.files = [array]$files.Name
}
}
catch {
echo 'xxx'
}
$hashtable.directories = #{}
foreach($directory in $directories) {
$element = Split-Path $directory.fullname -Leaf
$hashtable.directories.$element = build-json -fullPath $directory.FullName -hashtable $hashtable
}
return $hashtable
}
$sourcefolder = 'E:\xxx\Software\yyy\python392'
$result3 = build-json -fullPath $sourcefolder -hashtable #{}

Yes, it's possible.
You will need to write a little recursive function which walks through the dictionaries and collects the information for you.
You can use a hashtable (#{}) or a PowerShell object, but a hashtable is preferable.
Get-ChildItem and ConvertTo-Json are your friends

Related

How to insert bulk document into elastic using PowerShell?

I am trying to read the data from CSV file which has 2200000 records using PowerShell and storing each record in JSON file as NDJSON format.
Total document in JSON file is 2200000 (200 MB)
Sample JSON Data
{"index":{}}
{"ip-address":"1.5.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8325021+05:30"}
{"index":{}}
{"ip-address":"243.11.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
{"index":{}}
{"ip-address":"253.11.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
{"index":{}}
{"ip-address":"39.24.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
{"index":{}}
{"ip-address":"163.24.0.1","is-vpn":"true","#timestamp":"2022-12-01T18:59:48.8853225+05:30"}
Code
function Get-IPDataPath
{
$dbFilePath = Get-ChildItem -Path $rootDir -Filter "IP2*.CSV" | ForEach-Object FullName | Select-Object -First 1
Write-Host "file path - $dbFilePath"
$dbFilePath # implicit output
}
function Convert-NumberToIP
{
param(
[Parameter(Mandatory=$true)][string]$number
)
[Int64] $numberInt = 0
if( [Int64]::TryParse( $number, [ref] $numberInt ) ) {
if( ($numberInt -ge 0) -and ($numberInt -le 0xFFFFFFFFl) ) {
# Convert to IP address like '192.168.23.42'
([IPAddress] $numberInt).ToString()
}
}
# In case TryParse() returns $false or the number is out of range for an IPv4 address,
# the output of this function will be empty, which converts to $false in a boolean context.
}
function Insert-Document
{
param(
[Parameter(Mandatory=$true)][string]$indexName,
[Parameter(Mandatory=$true)][object]$filePath
)
$url = "https://esdev2:9200/naveen/_doc/_bulk"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($cred))
$headers = #{
"Authorization" = "Basic $encodedCreds"
"Transfer-Encoding" = "chunked"
"Content-Type" = "application/x-ndjson"
}
$content = Get-Content -Path $filepath
$response = Invoke-WebRequest -Uri $url -Method POST -Body $content -Headers $Headers -ContentType 'application/x-ndjson'
if($response.StatusCode -eq 201)
{
Write-Host "Documents added successfully"
}
else
{
Write-Host $response
Write-Host "$($response.content)"
throw "Trouble adding document: $_"
}
}
$dbFilePath = Get-IPDataPath
$outputFile = Join-Path -Path $rootDir -ChildPath "output.json"
Write-Host "Converting CSV file $dbFilePath to $outputFile"
$object = [PSCustomObject]#{
'ip-address' = ''
'is-vpn' = 'true'
'#timestamp' = ''
}
# Enclose foreach loop in a script block to be able to pipe its output to Set-Content
$count = 0
& {
foreach( $item in [Linq.Enumerable]::Skip( [IO.File]::ReadLines( $dbFilePath ), 1 ) )
{
$row = $item -split ','
$ipNumber = $row[0].trim('"')
if( $ip = Convert-NumberToIP -number $ipNumber )
{
$index = [PSCustomObject]#{"index" = [PSCustomObject]#{}} | ConvertTo-Json -Depth 10 -Compress
$object.'ip-address' = $ip
$object.'#timestamp' = (Get-Date).ToString('o')
#$index | ConvertTo-Json -Depth 100 -Compress
$document = $object | ConvertTo-Json -Depth 100 -Compress
if($count -lt 25)
{
Add-Content -Path $outputFile $index
Add-Content -Path $outputFile $document
$count++
}
else
{
$count = 0
BulkInsert-Document -indexName $indexName -filePath $outputFile
Clear-Content -Path $outputFile
}
}
}
} | Set-Content -Path $outputFile
Write-Host "Inserting document in elastic"
Insert-Document -indexName $indexName -filePath $outputFile
But I am getting Error: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk
request must be terminated by a newline
[\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be
terminated by a newline [\n]"},"status":400}
Though I have new line at the end of JSON file, Could you please help how to insert all the document.

Replace parameters in JSON with given Values from .csv

I have a JSON-File where I want to replace several values with a placeholder.
So i made a .csv with the parameters to replace. line[0] (if existing) is the path in the file, line[1] the element, line[2] the placeholder.
journeys.legs.origin.properties.downloads;url;placeholder;
;url;placeholder;
download;url;placeholder;
;psFileName;placeholder;
;serverTime;placeholder;
;calcTime;placeholder;
now I defined the following functions, to get the file, read the csv and replace the stuff.
$storage = "D:\Service\test.json"
$parameters2replace = "D:\Service\parameters2replace.csv"
function Get-JSONProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
function setParameter(){
foreach ($parameter in Get-Content $parameters2replace){
$line=$parameter.split(";")
$path = $line[0]
$elementName = $line[1]
$newValue = $line[2]
replaceElement $path $elementName $newValue
}
}
function replaceElement($path, $elementName, $newValue){
ForEach($JSONPath in Get-JSONProperty $JSON $path){
if (!$line[0]){
if($JSON.$elementName){
$JSON.$elementName = $newValue
}
}
else{
if($JSON.$path.$elementName){
echo $JSON.$path.$elementName
$JSON.$path.$elementName = $newValue
}
}
}
$JSON | ConvertTo-Json -depth 32| set-content $storage
}
$JSON = Get-Content $storage -raw | ConvertFrom-Json
setParameter
My problem now is, that the following if-argument won't work with the $path variable. If i put it in hardcoded it works just fine.
if($JSON.$path.$elementName)
I hope i could make everything clear, this was my first post.
$JSON.$path makes PowerShell look for a property with the literal name 'journeys.legs.origin.properties.downloads'. You need to split the string into it's individual parts and iterate (or recurse) through the path:
function Get-JSONPropertyValue {
param(
$JSON,
[string]$Path,
[string]$Name
)
$object = $JSON
foreach($propName in $Path.Split('.')){
$object = $object.$propName
}
return $object.$Name
}
function Set-JSONPropertyValue
{
param(
$JSON,
[string]$Path,
[string]$Name,
$Value
)
$object = $JSON
foreach($propName in $Path.Split('.')){
$object = $object.$propName
}
$object.$Name = $Value
}
Now you can do:
if($value = Get-JSONPropertyValue $JSON -Path $path -Name $elementName){
echo "Old value: $value"
Set-JSONPropertyValue $JSON -Path $path -Name $elementName -Value $newValue
}

ssis package importing data from email attachments to excel

I have got 100 plus email in a folder and all of them have got an attachment. I want to create a package which will copy the data from all email attachments in one excel sheet and also add a column which will have the received dates of those emails against the data.Can any one guide me how to do it with help of ssis package.
Not ssis but it should produce the desired result. Just change the path on the last line.
function Get-Attachment
{
[CmdletBinding()]
Param
(
[Parameter(ParameterSetName="Path", Position=0, Mandatory=$True)]
[String]$Path,
[Parameter(ParameterSetName="LiteralPath", Mandatory=$True)]
[String]$LiteralPath,
[Parameter(ParameterSetName="FileInfo", Mandatory=$True, ValueFromPipeline=$True)]
[System.IO.FileInfo]$Item
)
Begin
{
remove-item $Path\attachments.xlsx
$outlook = New-Object -ComObject Outlook.Application
$excel = New-Object -ComObject Excel.Application
$excel.visible = $true
$workbook = $excel.Workbooks.add()
$s1 = $workbook.Sheets.add()
$s1.name = "Attachments"
$cells= $s1.Cells
$s1.range("A1:A1").font.bold = "true"
$s1.range("A1:A1").cells="RecDate"
$s1.range("B1:B1").cells="Data"
$s1.range("B1:B1").font.bold = "true"
}
Process
{
switch ($PSCmdlet.ParameterSetName)
{
"Path" { $files = Get-ChildItem -Path $Path }
"LiteralPath" { $files = Get-ChildItem -LiteralPath $LiteralPath }
"FileInfo" { $files = $Item }
}
$row=2
$col=2
$files | % {
$msgFn = $_.FullName
if ($msgFn -notlike "*.msg") {
return
}
$msg = $outlook.CreateItemFromTemplate($msgFn)
$rdate = $msg.ReceivedTime
$msg.Attachments | % {
$attFn = $msgFn -replace '\.msg$', " - Attachment - $($_.FileName)"
if (Test-Path -literalPath $attFn) {
return
}
$_.SaveAsFile($attFn)
Get-ChildItem -LiteralPath $attFn
$d = ( Get-Content -Path $attFn -Raw)
$cells.item($row,$col)=$d.ToString()
$cells.item($row,1) = $rdate
$row++
}
$s1.range("A1:B1").EntireColumn.autofit() | out-Null
$s1.range("A1:B1").EntireColumn.WrapText = $false | out-Null
}
}
End
{
$workbook.SaveAs("$Path\attachments.xlsx")
}
}
Get-Attachment -Path C:\Users\Administrator\pathtodir\

Sharepoint 2013 SelectedRefinementControlsJson throwing error

I am trying to update the sharepoint 2013 refiner webpart thru the below powershell script:
$spweb = Get-SPWeb "http://da-server/search/"
$page = $spweb.GetFile("Pages/results.aspx")
$page.CheckOut()
# Find the Refinement web part
$webPartManager = $spweb.GetLimitedWebPartManager("http://da- server/search/Pages/results.aspx", [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)
$rwp = $webpartmanager.webparts | ? { $_.title -eq 'Refinement' }
# Refiners are updated by changing the JSON
$j = $rwp.SelectedRefinementControlsJson | ConvertFrom-Json
$j.refinerConfigurations | % { if ($_.propertyName -eq 'LastModifiedTime') { $_.displayName = 'Mod Time'; }
**$rwp.SelectedRefinementControlsJson = ConvertTo-Json $j -Compress** This line throwing the error.
# Save our changes
$webpartmanager.SaveChanges($rwp) # save changes to webpart
$page.CheckIn('Changed last mod refiner')
$page.Publish('Changed last mod refiner')
The above highlighted line 10 throwing the below error:
Exception setting "SelectedRefinementControlsJson": No parameterless constructor defined for type of 'Micrososft.Office.Server.Search.WebControls.RefinementControl[]'"
Try ConvertTo-Json $j -Depth 3
Out of box, Modified By property has more than one aliases and it's a nested level. By default ConvertTo-Json only goes 2 level. So you have to use -depth to convert it properly.
I guess you have in one or more refiners only one alias defined. In this case the PS ConvertTo-Json function converts the array to a string. More details found here https://superuser.com/questions/414650/why-does-powershell-silently-convert-a-string-array-with-one-item-to-a-string
As I struggled with the same issue here is my solution
Function ConvertTo-Json {
param($inputObject)
if ($inputObject -eq $null) { "null" }
else {
switch ($inputObject.GetType().Name) {
"String" { '"' + $inputObject +'"' }
"Boolean" {
if($inputObject){
"true"
}
else {
"false"
}
}
"Object[]" {
$items = #()
$inputObject | % {
$items += ConvertTo-Json $_
}
$ofs = ","; "[" + [string]$items + "]"
}
"PSCustomObject" {
$properties = #()
$inputObject | Get-Member -MemberType *Property | % {
$properties += '"'+ $($_.Name) + '":' + $(ConvertTo-Json $inputObject.($_.Name))
}
$ofs = ","; "{" + [string]$properties + "}"
}
default { $inputObject }
}
}
}
Now it can be used as before, but without -Compress parameter
$rwp.SelectedRefinementControlsJson = ConvertTo-Json $j

Powershell Recursion with Return

I am trying to write a recursive function that will return information in an array, however when I put a return statement into the function it misses certain entries.
I am trying to recursively look through a specified depth of folders getting the acl's associated with the folder. I know getChildItem has a recurse option, but I only want to step through 3 levels of folders.
The excerpt of code below is what I have been using for testing. When getACLS is called without a return statement (commented out below) the results are:
Folder 1
Folder 12
Folder 13
Folder 2
When the return statement is used I get the following output:
Folder 1
Folder 12
So it looks like the return statement is exiting out from the recursive loop?
The idea is that I want to return a multidimensional array like [folder name, [acls], [[subfolder, [permissions],[[...]]]]] etc.
cls
function getACLS ([string]$path, [int]$max, [int]$current) {
$dirs = Get-ChildItem -Path $path | Where { $_.psIsContainer }
$acls = Get-Acl -Path $path
$security = #()
foreach ($acl in $acls.Access) {
$security += ($acl.IdentityReference, $acl.FileSystemRights)
}
if ($current -le $max) {
if ($dirs) {
foreach ($dir in $dirs) {
$newPath = $path + '\' + $dir.Name
Write-Host $dir.Name
# return ($newPath, $security, getACLS $newPath $max ($current+1))
# getACLS $newPath $max ($current+1)
return getACLS $newPath $max ($current+1)
}
}
} elseif ($current -eq $max ) {
Write-Host max
return ($path, $security)
}
}
$results = getACLS "PATH\Testing" 2 0
The problem was the location of the return. I had it inside the foreach loop, meaning it was trying to return multiple times in the one function. I moved it outside the foreach, into the if statement instead.
function getACLS ([string]$path, [int]$max, [int]$current) {
$dirs = Get-ChildItem -Path $path | Where { $_.psIsContainer }
$acls = Get-Acl -Path $path
$security = #()
$results = #()
foreach ($acl in $acls.Access) {
$security += ($acl.IdentityReference, $acl.FileSystemRights)
}
if ($current -lt $max) {
if ($dirs) {
foreach ($dir in $dirs) {
$newPath = $path + '\' + $dir.Name
$next = $current + 1
$results += (getACLS $newPath $max $next)
}
} else {
$results = ($path, $security)
}
return ($path, $security, $results)
} elseif ($current -eq $max ) {
return ($path, $security)
}
}
In recursion, I would only use a return statement where I needed to end the recursion - just for clarity. I've done a good bit of recursion in PowerShell and it works well. However you need to remember that PowerShell functions do behave differently. The following:
return 1,2
is equivalent to:
1,2
return
In other words, anything you don't capture in a variable or redirect to a file (or $null) is automatically considered output of the function. Here's a simple example of a working, recursive function:
function recursive($path, $max, $level = 1)
{
$path = (Resolve-Path $path).ProviderPath
Write-Host "$path - $max - $level"
foreach ($item in #(Get-ChildItem $path))
{
if ($level -eq $max) { return }
recursive $item.PSPath $max ($level + 1)
}
}
recursive ~ 3
Update: I am leaving the first answer as is and adding the new code here. I see that there are multiple issues with your code. here is the updated one.
cls
function getACLS ([string]$path, [int]$max, [int]$current) {
$dirs = Get-ChildItem -Path $path | Where { $_.psIsContainer }
$acls = Get-Acl -Path $path
$security = #()
foreach ($acl in $acls.Access) {
$security += ($acl.IdentityReference, $acl.FileSystemRights)
}
if ($current -lt $max) {
if ($dirs) {
foreach ($dir in $dirs) {
$newPath = $dir.FullName
$security
getACLS $newPath $max ($current+1)
}
}
} elseif ($current -eq $max ) {
Write-Host max
return $security
}
}
$results = getACLS "C:\Scripts" 2 0
If you see above, I am not using return. I just throw the object from the GetACLs function. Also, I modified it to return on $security for testing purpose. I can see the all ACLs in $results. I changed the first if condition to if ($current -lt $max). It should not be if ($current -le $max).
Let me know if this what you are looking for. I can continue to optimize this.
==========================================OLD=============================================
Return will exit the function.
I am not providing the complete solution here but want to give you an idea about how this can be changed.
You can use PS Custom object to capture the information you need. For example,
function GetItem {
$itemsArray = #()
Get-ChildItem C:\Scripts | ForEach-Object {
$itemsObject = New-Object PSObject
Add-Member -InputObject $itemsObject -MemberType NoteProperty -Name "FullName" -Value $_.FullName
Add-Member -InputObject $itemsObject -MemberType NoteProperty -Name "Name" -Value $_.Name
$itemsArray += $itemsObject
}
return $itemsArray
}
This way you can return the object once it is completely built with the information you need.
I found none of these solutions did what I need to, which was to have an array with the results of the whole recursive function. Since we can't initialise the array inside the function, otherwise it is re-initialised every time recursion is used, we have to define it outside and use global:
# Get folder contents recursively and add to an array
function recursive($path, $max, $level = 1)
{
Write-Host "$path" -ForegroundColor Red
$global:arr += $path
foreach ($item in #(Get-ChildItem $path))
{
if ($level -eq $max) { return }
if ($item.Length -eq "1") # if it is a folder
{
$newpath = "$path\$($item.Name)"
recursive $newpath $max ($level + 1)
}
else { # else it is a file
Write-Host "$path\$($item.Name)" -ForegroundColor Blue
$global:arr +="$path\$($item.Name)"
}
}
}
$arr = #() # have to define this outside the function and make it global
recursive C:\temp\test2 4
write-host $arr.count