Merge two CSV files into one - csv

I have two CSV files that don't have any unique identifiers. The number of rows of both files is always the same. I want to merge them as it is (the 2nd CSV as additional columns into the 1st CSV).
Content of file1.csv:
Server,Info
server1,item1
server1,item2
server1,item3
server2,item1
server2,item2
server2,item3
server3,item1
server3,item2
server3,item3
Content of file2.csv:
Items,ColumnA,ColumnB,ColumnC
item#1:,aa,jj,ss
item#2:,bb,kk,tt
item#3:,cc,ll,uu
item#1:,dd,mm,vv
item#2:,ee,nn,ww
item#3:,ff,oo,xx
item#1:,gg,pp,yy
item#2:,hh,qq,zz
item#3:,ii,rr,ab
Expecting output of csv file:
Server,Info,Items,ColumnA,ColumnB,ColumnC
server1,item1,item#1:,aa,jj,ss
server1,item2,item#2:,bb,kk,tt
server1,item3,item#3:,cc,ll,uu
server2,item1,item#1:,dd,mm,vv
server2,item2,item#2:,ee,nn,ww
server2,item3,item#3:,ff,oo,xx
server3,item1,item#1:,gg,pp,yy
server3,item2,item#2:,hh,qq,zz
server3,item3,item#3:,ii,rr,ab
I have search intensively in the net, but couldn't find any solution... Would really appreciate if someone can provide me some answers here...


You can use the Get-Content cmdlet to read both CSV files, iterate over it using a for-loop and use a format string to join them together. Finally, write the new file back using the Out-File cmdlet:
$file1 = Get-Content 'Path_to_file1.csv'
$file2 = Get-Content 'Path_to_file2.csv'
$content = for ($i = 0; $i -lt $file1.Length; $i++)
{
'{0},{1}' -f $file1[$i].Trim(), $file2[$i].Trim()
}
$content | out-file 'Path_to_file3.csv'

Related

Powershell add header to blank csv

I have a process that I am trying to fully automated and have hit upon a stumbling block.
The process runs through a number of SQL queries and then outputs these results to different named CSV files.
Where my issue lies is that where no results are returned in the query to the dataset when I export the data (which I must do to satisfy auditors) there is no header data written to the CSV file.
What I need to do therefore is IF the dataset contains 0 rows then to simply export a CSV file containing the headings "Client", "Balance", "Account".
Where as if there is data within the table then the process can continue as it currently does.
What I am is unsure how the hell this can be achieved...
I am currently muting whether or not I will have to export the data and then write a loop to delete any lines from the CSV containing for example "Client" then re-import the CSV's add the column heading that I want and export it again. As this is messy, ideally I would like to keep away from this..
Code is
$SqlCmd.CommandTimeout=$timeout;
$SqlCMD.CommandText = $034CASHQUERY;
$SqlCmd.Connection = $SqlConnection;
## - Extract Data and build sql data object
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter;
$SqlAdapter.SelectCommand = $SqlCmd;
$DataSet = New-Object System.Data.DataSet;
$SqlAdapter.Fill($DataSet);
$DataSetTable = $DataSet.Tables["Table"];
IF($DataSet.Tables["Table"]){
$isnotnull
$DataSet.Tables["Table"] | Export-Csv $034CASHOUT -NoTypeInformation
echo "CL2 Exported"
}
else {
}
Has anyone come across this before or aware of how to iterate through the issue?

You could create a .csv file and add the header text manually.
$headerText = ('"Client","Balance","Account"' + "`n")
New-Item $034CASHOUT | Add-Content -value $headerText
Also, I think your IF condition may be a bit wonky. Are you looking for:
IF($DataSet.Tables["Table"] -isnot $null)
All together:
IF($DataSet.Tables["Table"] -isnot $null){
$DataSet.Tables["Table"] | Export-Csv $034CASHOUT -NoTypeInformation
echo "CL2 Exported"
}
else {
$headerText = ('"Client","Balance","Account"' + "`n")
New-Item $034CASHOUT | Add-Content -value $headerText
}
This is probably not the best way to do it, but I think this is what you described.

Can you please provide more information on the data you're processing? I understand if there is no data you want "Client", "Balance", "Account".
When there is data present, will it also be in these 3 columns?
If so this is your answer:
$DataSet.Tables["Table"] |
Select-Object Client,Balance,Account |
Export-Csv $034CASHOUT -NoTypeInformation

Quickly Split CSV line that is Comma Delimited with quotes around Strings in Powerhell

I have large CSV files that 0.5-2gb+ files I am trying to import with Powershell.
Data looks like so:
Name, Date, Value
"Joe, John", 2016-08-01, "value"
"Smith, Jane", 2016-08-01, "value"
...
I have this function
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
$reader = new-object System.IO.StreamReader($csv)
while (($line = $reader.ReadLine()) -ne $null) {
# Use RegEx to only split on (,) outside quotes and remove quoted strings
$row = ($line -split ',(?=(?:[^"]|"[^"]*")*$)').Replace("`"","")
# Row Indicator
$i++;
if (($i % 50000) -eq 0) {
Write-Host "$i rows have been processed in $($elapsed.Elapsed.ToString())."
}
}
Splitting the line by a comma "," works perfect as I get ~16K a second, but I need to only split outside of any quotes, so I implemented the regular expression, however the performance tanks to 900 rows a second.
I am looking for a more efficient way to loop through a CSV file that is comma delimited but has commas in the quotes that need to be excluded.

Import-Csv, as noted in the comments above, does not load everything into memory unless you ask it to. Like the example in the question it implements a stream reader and pushes the content it's read off to the output pipeline.
You will see significant memory usage if you do something like this:
$var = Import-Csv thefile.csv
After all, the content of the CSV has to go somewhere.
Whereas if you do something with the output pipeline there's less impact. e.g.
Import-Csv thefile.csv | ForEach-Object {
Do-Something
}
Finally, Import-Csv really doesn't work for you I have a CSV reader class along with a side-by-side implementation of Import-Csv called Indented.Text.Csv on github. This implementation provides a public class with a number of features I needed so I could process CSV files very quickly.

PowerShell: Get 2 strings into a hashtable and out to .csv

PowerShell newbie here,
I need to:
Get text files in recursive local directories that have a common string, students.txt in them.
Get another string, gc.student="name,name" in the resulting file set from #1 and get the name(s).
Put the filename from #1, and just the name,name from #2 (not gc.student="") into a hashtable where the filename is paired with its corresponding name,name.
Output the hashtable to an Excel spreadsheet with 2 columns: File and Name.
I've figured out, having searched and learned here and elsewhere, how to output #1 to the screen, but not how to put it into a hashtable with #2:
$documentsfolder = "C:\records\"
foreach ($file in Get-ChildItem $documentsfolder -recurse | Select String -pattern "students.txt" ) {$file}
I'm thinking to get name in #2 I'll need to use a RegEx since there might only be 1 name sometimes.
And for the output to Excel, this: | Export-Csv -NoType output.csv
Any help moving me on is appreciated.

I think this should get you started. The explanations are in the code comments.
# base directory
$documentsfolder = 'C:\records\'
# get files with names ending with students.txt
$files = Get-ChildItem $documentsfolder -recurse | Where-Object {$_.Name -like "*students.txt"}
# process each of the files
foreach ($file in $files)
{
$fileContents = Get-Content $file
$fileName = $file.Name
#series of matches to clean up different parts of the content
#first find the gc.... pattern
$fileContents = ($fileContents | Select-String -Pattern 'gc.student=".*"').Matches.Value
# then select the string with double quotes
$fileContents = ($fileContents | Select-String '".*"').Matches.Value
# then remove the leading and trailing double quotes
$fileContents = $fileContents -replace '^"','' -replace '"$',''
# drop the objects to the pipeline so that you can pipe it to export-csv
# I am creating custom objects so that your CSV headers will nave nice column names
Write-Output [pscustomobject]#{file=$fileName;name=$fileContents}
} | Export-Csv -NoType output.csv

Find and Replace many Items with Powershell from Data within a CSV, XLS or two txt documents

So I recently have found the need to do a find and replace of mutliple items within a XML document. Currently I have found the code below which will allow me to do multiple find and replaces but these are hard coded within the powershell.
(get-content c:\temp\report2.xml) | foreach-object {$_ -replace "192.168.1.1", "Server1"} | foreach-object {$_ -replace "192.168.1.20", "RandomServername"} | set-content c:\temp\report3.xml
Ideally instead of hard coding the value I would like to find and replace from a list, ideally in a CSV or and XLSX. Maybe two txt file would be easier.
If it was from a CSV it could grab the value to find from A1 and the value to replace it with from B1 and keep looping down until the values are empty.
I understand I would have to use the get-content and the for each command I was just wondering if this was possible and how to go about it/ if anybody could help me.
Thanks in advance.
SG

#next line is to clear output file
$null > c:\temp\report3.xml
$replacers = Import-Csv c:\temp\replaceSource.csv
gc c:\temp\aip.xml | ForEach-Object {
$output = $_
foreach ($r in $replacers) {
$output = $output -replace $r.ReplaceWhat, $r.ReplaceTo
}
#the output has to be appended, not to rewrite everything
return $output | Out-File c:\temp\report3.xml -Append
}
Content of replaceSource.csv looks like:
ReplaceWhat,ReplaceTo
192.168.1.1,server1
192.168.1.20,SERVER2
Note the headers

Powershell - find file by 'file name' and rename based on CSV

I have a set of files (OldName) in a Windows directory that I would like to rename (NewName) based on the following CSV file:
OldName,NewName
Sources/texas play_PGC_Cpgc_entryPoint_Cbp_1.f4v,01 Texas Play.f4v
Sources/texas play_PGC_Cpgc_entryPoint_Dbp_1.f4v,02 First Song.f4v
Sources/texas play_PGC_Cpgc_entryPoint_Ebp_1.f4v,03 Yellow Rose.f4v
I'm not sure how to loop thru the CSV file... finding each file and replacing.
Any thoughts would be appreciated.

First Import Your CSV file into powershell
$AnyVariableName = Import-Csv "$env:USERPROFILE:\Desktop\directoryname.txt"
Note: In my example, the path to the CSV file is on my desktop, but it may be different in yours.
Then use a foreach loop rename the items
foreach ($objName in $AnyVariableName){
Rename-Item $objName.OldName $objName.NewName
}

One way to do it is to create two lists and loop though each of them. The CSV file will be a reference list, so we'll grab the contents and convert it from CSV then store it in a variable
$CSVRef = Get-Content "C:\Path\To\CSV.csv" | ConvertFrom-CSV
Then we'll get the list of files who's names you want to change, and loop through each file. From inside the loop you can run another loop to find the current name in your reference list, and then change it to the new name.
Get-ChildItem "C:\path\to\f4v\files" -Filter *.f4v | ForEach-Object {
#Grab the current item in a variable to access it within the second loop
$CurrentFile = $_
$CSVRef | ForEach-Object {
if ($CurrentFile.Name -ilike $_.OldName) {
Rename-Item $CurrentFile.FullPath $_.NewName
}
}
}
So during the second loop we try to compare the file name with every "OldName" item in the CSV file list. If the OldName matches somewhere in the current file we're looping through then we run Rename-Item and provide it the NewName. It should automatically rename the file.

Combining both examples works great
$CSVRef = Import-Csv "C:\Temp\Filename.txt"
Get-ChildItem "C:\Temp\FileFolder" -Filter *.pdf | ForEach-Object {
$CurrentFile = $_
ForEach ($objName in $CSVRef) {
if ($CurrentFile.Name -ilike $objName.OLDNAME) {
Rename-Item $CurrentFile.FullName $objName.NEWNAME
}
}
}