Import-Csv TrimEnd Column Header - csv

I need import a CSV and run it through a foreach loop. I want to trim the end on the column header DeviceName to avoid any potential issues. I have tried the following but it is not working as expected.
$Import = Import-CSV $csv
foreach ($i in ($import.DeviceName).TrimEnd())
{do something}
Any help? Thank you!

If you need to change both the header and the content in the column for devicename which has spaces I have come up with this forgiving code.
$csvData = import-csv $csv
$properties = $csvData[0].psobject.Properties.name
$csvHeader = "`"$(($properties | ForEach-Object{$_.Trim()}) -join '","')`""
$deviceHeader = $properties -match "DeviceName"
$csvHeader
$csvHeader | Set-Content $file
$csvData | ForEach-Object{
$_.$deviceHeader = ($_.$deviceHeader).trim()
$_
} | ConvertTo-Csv -NoTypeInformation | Select-Object -Skip 1 | Add-Content $file
What this does is read in the CSV like normal. Parse the property names of the object in the order they appear. We find the one that has DeviceName no matter how many spaces (if there is more that one you could have a problem). Keep that so we can use it to call the correct property of each "row".
Export the new cleaned header to the file. Then we go through each "row" removing all the leading and trailing space from the DeviceName. Once that is done write back the CSV to the original file.

The best solution would be to tell the other team to fix their generation procedure. However, if for some reason that's not an option, I'd recommend pre-processing the file before you import it as a CSV.
$filename = 'C:\path\to\your.csv'
(Get-Content $filename -Raw) -replace '^(.*DeviceName)[ ]*(.*)', '$1$2' |
Set-Content $filename
Reading the file as a single string (-Raw) and anchoring the expression at the beginning of the string (^) ensures that only the column title is replaced.
For large input files you may want to consider a different approach, though, since the above reads the entire file into memory before replacing the first line.
$infile = 'C:\path\to\input.csv'
$outfile = 'C:\path\to\output.csv'
$firstLine = $true
Get-Content $infile | % {
if ($firstLine) {
$_ -replace '(DeviceName)[ ]*', '$1'
$firstLine = $false
} else {
$_
}
} | Set-Content $outfile
Thinking about it some more and taking inspiration from a comment to #Zeek's answer, you could also extract the headers first and then convert the rest of the file.
$infile = 'C:\path\to\input.csv'
$outfile = 'C:\path\to\output.csv'
$header = (Get-Content $infile -First 1) -split '\s*,\s*'
Get-Content $infile |
select -Skip 1 |
ConvertFrom-Csv -Header $header |
Export-Csv $outfile -NoType

Is this all you're trying to do? This will give you a collection of objects imported from your csv file but trim the end of the DeviceName property on each object.
$items = Import-CSV -Path $csv
$items.ForEach({ $_.DeviceName = $_.DeviceName.TrimEnd() })

Related

Powershell ForEach-Object column variables

Kind of have a strange problem. I have a large JSON file that needs to be processed. Based on another question I need to stream the file since it will otherwise gets me problems because of memory: JSON Powershell memory issue
What I have is this:
get-content -Path largefile.json | ForEach-Object {
$row = $_ = $_.TrimStart('[').TrimEnd(']')
if ($_) { $_ | Out-String | ConvertFrom-Json }
New-Item -Path $($Row.Id).txt
Set-Content -Path $($Row.Id).txt -Value ($row.Body)
}
I can easily do $row to publish the last processed row in the Largefile.json. I want to create a file with the name of the Id in the row that is currently processed and add the body column to the file. But when I want to show a specific column using $row.Id, unfortunately this shows up empty.
The structure of the Largefile.json is as followed:
[{"Id":"1","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data1"}
{"Id":"2","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data2"}
{"Id":"3","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data3"}
{"Id":"4","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data4"}
{"Id":"5","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data5"}
]
The end result should be that I have 5 files:
1.txt - Value inside the file should be: data1
2.txt - Value inside the file should be: data2
3.txt - Value inside the file should be: data3
4.txt - Value inside the file should be: data4
5.txt - Value inside the file should be: data5
I use Powershell 7.1.3
Is there any way that I can use $row.Id and $row.ParentId just like a regular ForEach would do?
thanks for your help.
It seems to me that this is what you're looking for:
Get-Content largefile.json | ForEach-Object {
$row = $_.TrimStart('[').TrimEnd(']') | ConvertFrom-Json
if ($null -ne $row) {
Set-Content -Path ($row.Id) -Value ($row.Body)
}
}
I am still not sure what you expect as an outcome.
But I think you want to do this:
#'
[{"Id":"1","ParentId":"parent1","Name":"1.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data1"}
{"Id":"2","ParentId":"parent2","Name":"2.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data2"}
{"Id":"3","ParentId":"parent3","Name":"3.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data3"}
{"Id":"4","ParentId":"parent4","Name":"4.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data4"}
{"Id":"5","ParentId":"parent5","Name":"5.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data5"}
]
'# | Set-Content .\largefile.json
Get-Content .\largefile.json | ForEach-Object {
$_ = $_.TrimStart('[').TrimEnd(']')
If ($_) {
$Row = ConvertFrom-Json $_
Set-Content -Path ".\$($Row.Name)" -Value $Row.Body
}
}
The question has many errors. Assuming the json has the missing commas in, I would do it this way, if I understand the question. This should work with the new updates to the question. I also have a more unusual solution involving streaming json with jq here: Iterate though huge JSON in powershell Json streaming support may be added later: ConvertFrom-JSON high memory consumption #7698
[{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
]
get-content -Path largefile.json | ForEach-Object {
$_ = $_.TrimStart('[').TrimEnd(']').TrimEnd(',')
if ($_) {
$row = $_ | ConvertFrom-Json
Set-Content -Path ($Row.Id + '.txt') -Value $row.Body
}
}
get-content ID.txt
*******
As others already explained, your json example is invalid.
However, since this is a huge file to process, you can use switch for this.
switch -Regex -File D:\Test\largefile.json {
'"Id":"(\d+)".*"Body":"(\w+)"' {
Set-Content -Path ('D:\Test\{0}.txt' -f $matches[1]) -Value $matches[2]
}
}
Results using your example would be 5 files called 1.txt .. 5.txt, each having a single line data1 .. data5

Unable to combine all csv files using powershell

I would like to combine all the csv files in my local folder but it shows empty results. I am trying to take the header of the first file and skip all the headers in the rest of the files in the folder and join them.
get-childItem "C:\Users\*.csv" | foreach {[System.IO.File]::AppendAllText
("C:\Users\finalCSV.csv", [System.IO.File]::ReadAllText($_.FullName))}
$getFirstLine = $true
get-childItem "C:\Users\*.csv" | foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "C:\Users\finalCSV.csv" $linesToWrite
}
My end result is that when I open finalCSV.csv it shows no results.
I think you are trying to overwork your solution. Just use Import-Csv and append to an array. Something like this:
$a = #(); ls *.csv | % {$a += (Import-Csv $_.FullName)}; $a
Works even if the columns are in a different order.

Find and Replace many Items with Powershell from Data within a CSV, XLS or two txt documents

So I recently have found the need to do a find and replace of mutliple items within a XML document. Currently I have found the code below which will allow me to do multiple find and replaces but these are hard coded within the powershell.
(get-content c:\temp\report2.xml) | foreach-object {$_ -replace "192.168.1.1", "Server1"} | foreach-object {$_ -replace "192.168.1.20", "RandomServername"} | set-content c:\temp\report3.xml
Ideally instead of hard coding the value I would like to find and replace from a list, ideally in a CSV or and XLSX. Maybe two txt file would be easier.
If it was from a CSV it could grab the value to find from A1 and the value to replace it with from B1 and keep looping down until the values are empty.
I understand I would have to use the get-content and the for each command I was just wondering if this was possible and how to go about it/ if anybody could help me.
Thanks in advance.
SG
#next line is to clear output file
$null > c:\temp\report3.xml
$replacers = Import-Csv c:\temp\replaceSource.csv
gc c:\temp\aip.xml | ForEach-Object {
$output = $_
foreach ($r in $replacers) {
$output = $output -replace $r.ReplaceWhat, $r.ReplaceTo
}
#the output has to be appended, not to rewrite everything
return $output | Out-File c:\temp\report3.xml -Append
}
Content of replaceSource.csv looks like:
ReplaceWhat,ReplaceTo
192.168.1.1,server1
192.168.1.20,SERVER2
Note the headers

Convert tab delimiter to semicolon

I have updated a piece of software for our T&A system, this produces a CSV file in tab-delimited format. The payroll software needs this in the older format which was semicolon-delimited. I have been in touch with both vendors and neither one has a way to accommodate the other so I need to convert the CSV file to suit the payroll software. I have tried to do this with PowerShell with mixed results.
First I tried
Import-Csv ".\desktop\new version.csv" -Delimiter `t |
Export-Csv ".\converted.csv" -NoTypeInf
which removed the tab delimiter but didn't do the ;. So I then tried
Import-Csv ".\desktop\new version.csv" -Delimiter `t |
Export-Csv ".\desktop\converted.csv" -NoTypeInformation -Delimiter ";"
which did convert it from tabbed to ;, but only for the headers. It totally ignored the rest of the data. I then tried a different approach and used
$path = ".\desktop\new.csv"
$outPath = ".\desktop\converted.csv"
Get-Content -path $path |
ForEach-Object {$_ -replace "`t",";" } |
Out-File -filepath $outPath
which formatted the file correctly, but put an extra empty row between each row of data. I'm not sure what I'm doing wrong!
I'm pretty sure you are having an encoding issue with your last example. Get-Content reads in as Ascii whereas Out-File defaults to Unicode. Either set the -Encoding on Out-File or just use Set-Content.
Get-Content -path $path |
ForEach-Object {$_ -replace "`t",";" } |
Set-Content -filepath $outPath
You could even trim this down a bit if need be.
(Get-Content -path $path) -replace "`t",";" | Set-Content -filepath $outPath
However your 2nd code example...
Import-Csv ".\desktop\new version.csv" -Delimiter `t | Export-Csv ".\desktop\converted.csv" -NoTypeInformation -Delimiter ";"
should have worked just fine to replacing the tabs to semicolons. If it is not working then I would think your source data has an issue.
About the source file
Based on comments the code above is creating a trailing column. Most likely reason for that is trailing tabs on each row that are being converted. If that is the case then a little more manipulation would be required. Easier to use the foreach loop in this case.
Get-Content -path $path |
ForEach-Object {$_.Trim() -replace "`t",";" } |
Set-Content -filepath $outPath
That would remove the last tab/whitespace of each line. There is a potential enormous caveat doing it this way though. I think it has the potential to drop data if you have empty columns on the end. However if those columns were already empty it should not matter as long as the header is formed well and the input program can account for this. Else you are looking at reading in the file with Import-CSV and dropping the last column which can be done.
Here's a function I used to replace strings in text files like you're doing. This is assuming there's no tabs inside the text file other than those that are delimiting the columns. I'm assuming there's not. You can use it like this:
Find-InTextFile -FilePath C:\MyFile.csv -Find "`t" -Replace ';'
function Find-InTextFile
{
<#
.SYNOPSIS
Performs a find (or replace) on a string in a text file or files.
.EXAMPLE
PS> Find-InTextFile -FilePath 'C:\MyFile.txt' -Find 'water' -Replace 'wine'
Replaces all instances of the string 'water' into the string 'wine' in
'C:\MyFile.txt'.
.EXAMPLE
PS> Find-InTextFile -FilePath 'C:\MyFile.txt' -Find 'water'
Finds all instances of the string 'water' in the file 'C:\MyFile.txt'.
.PARAMETER FilePath
The file path of the text file you'd like to perform a find/replace on.
.PARAMETER Find
The string you'd like to replace.
.PARAMETER Replace
The string you'd like to replace your 'Find' string with.
.PARAMETER UseRegex
Use this switch parameter if you're finding strings using regex else the Find string will
be escaped from regex characters
.PARAMETER NewFilePath
If a new file with the replaced the string needs to be created instead of replacing
the contents of the existing file use this param to create a new file.
.PARAMETER Force
If the NewFilePath param is used using this param will overwrite any file that
exists in NewFilePath.
#>
[CmdletBinding(DefaultParameterSetName = 'NewFile')]
param (
[Parameter(Mandatory = $true)]
[ValidateScript({ Test-Path -Path $_ -PathType 'Leaf' })]
[string[]]$FilePath,
[Parameter(Mandatory = $true)]
[string]$Find,
[Parameter()]
[string]$Replace,
[Parameter()]
[switch]$UseRegex,
[Parameter(ParameterSetName = 'NewFile')]
[ValidateScript({ Test-Path -Path ($_ | Split-Path -Parent) -PathType 'Container' })]
[string]$NewFilePath,
[Parameter(ParameterSetName = 'NewFile')]
[switch]$Force
)
begin
{
if (!$UseRegex.IsPresent)
{
$Find = [regex]::Escape($Find)
}
}
process
{
try
{
foreach ($File in $FilePath)
{
if ($Replace)
{
if ($NewFilePath)
{
if ((Test-Path -Path $NewFilePath -PathType 'Leaf') -and $Force.IsPresent)
{
Remove-Item -Path $NewFilePath -Force
(Get-Content $File) -replace $Find, $Replace | Add-Content -Path $NewFilePath -Force
}
elseif ((Test-Path -Path $NewFilePath -PathType 'Leaf') -and !$Force.IsPresent)
{
Write-Warning "The file at '$NewFilePath' already exists and the -Force param was not used"
}
else
{
(Get-Content $File) -replace $Find, $Replace | Add-Content -Path $NewFilePath -Force
}
}
else
{
(Get-Content $File) -replace $Find, $Replace | Add-Content -Path "$File.tmp" -Force
Remove-Item -Path $File
Rename-Item -Path "$File.tmp" -NewName $File
}
}
else
{
Select-String -Path $File -Pattern $Find
}
}
}
catch
{
Write-Error -Message $_.Exception.Message
}
}
}

How can I combine fields in a .csv based off of a shared value in powershell?

I have two files in identical formats, one containing destination IP addresses and URLs, and one that contains only the destination IP addresses. I am attempting to write a powershell script to add the URL field from the first file to the second file for that row if the destination IP addresses are equal. Here is an example of the two files:
File Containing URLs:
Date;Time;Source;Destination;Port;User;URL
3/7/2016;0:00:07;168.254.25.6;10.0.1.27;80;jsmith;abcnet
File to add URLs to:
Date;Time;Source;Destination;Port;User;URL
3/7/2016;0:00:09;168.254.25.6;10.0.1.27;80;;
Whenever I run the code below, it appears to be caught in an infinite loop because it does not run to completion, but it throws no errors. My data set is thousands of lines long, but it works when I test it with a sample set that is only a few lines long.
$noURLs = Import-Csv C:\Path\to\noURLs.csv
$containsURLs = Import-Csv C:\Path\to\containsURLs.csv | Select-Object Destination, URL
$outputFile = "C:\Path\to\output.csv"
if(Test-Path $outputFile){
Remove-Item $outputFile
}
foreach($line in $noURLs){
$cpDest = $line.Destination
$destURL = $containsURLs | Where-Object {$_.Destination -eq $cpDest} | Select-Object -ExpandProperty URL | Select-Object -Unique
if($destURL -ne $null){
if( $destURL.Count -gt 1) {
$destURL = $destURL -join ';'
}
}
$line.URL = $destURL
}
$noURLs | Export-Csv $outputFile
I forgot to add a -unique switch to my select object, so for every one record in the first csv, it was looping through every single line of the second csv. Fixed code looks like this:
$noURLs = Import-Csv C:\Path\to\noURLs.csv
$containsURLs = Import-Csv C:\Path\to\containsURLs.csv | Select-Object -Unique Destination, URL
$outputFile = "C:\Path\to\output.csv"
if(Test-Path $outputFile){
Remove-Item $outputFile
}
foreach($line in $noURLs){
$cpDest = $line.Destination
$destURL = $containsURLs | Where-Object {$_.Destination -eq $cpDest} | Select-Object -ExpandProperty URL | Select-Object -Unique
if($destURL -ne $null){
if( $destURL.Count -gt 1) {
$destURL = $destURL -join ';'
}
}
$line.URL = $destURL
}
$noURLs | Export-Csv $outputFile -NoTypeInformation