I am having trouble in parsing the csv and deleting the rows as per the requirement.
My csv file format is not even, and there are many columns which are blank, But I want them to be considered as proper cells
csv has some rows (not fixed number of rows) before the actual header and at the bottom there may be a chance of having 2 lines. I want my csv to start from the header instead of the first line.
Import-Csv file.csv | Get-Member
this code only gives me two member types but csv has more than 20columns
csv example:
1,ab
x,33,455
blank line
x,12,134
some more lines
f,er,fsdvf
blank lines
headers start here, 20+columns
Thanks in advance
If you know the header (columns) you can always force Import-Csv to have these columns even if the first line of the file suggests otherwise:
Import-Csv file.csv -Header One, Two, Three, Four
Downside is that you end up with some objects that you probably don't need.
As alternative, you can find line with your header and use Get-Content/ConvertFrom-Csv combo:
$line = Select-String -Path file.csv -Pattern ^One |
ForEach-Object LineNumber
Get-Content -Path file.csv |
Select-Object -Skip ($line - 1) |
ConvertFrom-Csv
Regular expression should probably be more verbose, but overall that should be easiest that convincing application producing fake CSVs to the it proper.
Related
fairly new to powershell and I have given myself a bit of a challenge which I think should be possible, I'm just not sure about the best way around it.
We have a user who has a large number of columns in a csv (can vary from 20-50), rows can vary between 1 and 10,000. the data is say ClientName,Address1,Address2,Postcode etc.. (although these can vary wildly depending on the source of the data - external companies) This needs importing into a system using a pre-built routine which looks at the file and needs the database column headers as the csv headers. so say ClientDisplay,Ad_Line1,Ad_Line2,PCode etc..
I was thinking along the lines of either a generic powershell 'mapping' form which could read the headers from ExternalSource.csv and either a DatabaseHeaders.csv (or a direct sql query lookup) display them as columns in a form and then highlight one from each column and a 'link' button, once you have been through all the columns in ExternalSource.csv a 'generate csv' button which takes the mapped headers an appends the correct data columns from ExternalSource.csv
Am I barking up the wrong tree completely trying to use powershell? at the moment its a very time consuming process so just trying to make life easier for users.
Any advice appreciated..
Thanks,
Jon
You can use the Select-Object cmdlet with dynamic columns to shape the data into the form you need.
Something like:
Import-Csv -Path 'source.svc' |
Select-Object Id, Name, #{ Name='Ad_Line1'; Expression={ $_.Address1 } } |
Export-Csv -Path 'target.csv'
In this example, the code #{ Name='Ad_Line1'; Expression={ $_.Address1 } } is a dynamic column, that creates a column with name AD_Line1' and the value ofAddress1`
It is possible to read the column mappings from a file, you will have to write some code to read the file, select the properties and create the format.
A very simple solution could be to read the Select-Object part from another script file, so you can differentiate that part for each import.
A (simple, naive, low performant) solution could look like this (untested code):
# read input file
$input = Import-Csv -Path $inputFile
# read source, target name columns from mapping file
$mappings = Import-Csv -Path $mappingFile | Select Source, Target
# apply transformations
$transformed = $input
foreach($mapping in $mappings) {
# collect the data, add an extra column for each mapping
$transformed = $transformed | Select-Object *, #{ Name = $mapping.Target; Expression = { $_.$($mapping.Source) } }
}
#export transformed data
$transformed | Export-Csv -Path $outputFile
Alternatively; It is possible to convert the data into XML with Import-Csv | Export-CliXml, apply an Xslt template on the Xml to perform a transformation, and save the Xml objects into Csv again with Import-CliXml | Export-Csv.
See this blog by Scott Hansleman on how you can use XSLT with PowerShell.
I'm trying to cycle through a csv and replace any values in a column named Enabled from True to A.
Import-Csv .\test.csv | Where-Object {$_.Enabled -eq 'True'} --> what goes here to replace 'True' with 'A'?
Where-Object acts like a filter, so the columns that get passed to the rest of the pipeline will only be the ones where Enabled is True; which will prevent you from including the others in your output file (I'm assuming you want to have a complete file at the end).
So I would recommend using ForEach-Object and then modifying based on a condition inside there, but still passing each object through (modified or not):
Import-Csv .\test.csv | ForEach-Object {
if ($_.Enabled -eq 'True') {
$_.Enabled = 'A'
}
$_
} | Export-Csv .\test-modified.csv -NoTypeInformation
Briantist's answer works just fine. If you really wanted to get crazy with it you could create an Excel comobject, select the workbook/sheet, then select the "enabled" entire column, snip out empty cells/column header, then loop through and essentially do the same thing as what briantist said, although this way you can do things like add conditional formatting, etc. Just depends what all you are trying to do
I got multiple csv files, all with the same header, all in the same folder which I have to combine to create one large csv file. Since every .csv-file has the same header, I guess using it once should be enough.
All files look like this (also delimited by ','):
Header1,Header2,Header3
Data1, Data2, Data3
Data4, Data5, Data6
Can you help out? I'm not very comfortable with Powershell yet, tried out different codes but nothing really helped me out.
Thanks
if all csv's has the same columns, simply:
# Original CSV
$csv = import-csv c:\temp.csv
# 2nd CSV
$csv2 = import-csv c:\temp2.csv
# As simple as:
$csv += $csv2
If you want to import all CSV's in the current folder you can do something like the following:
Get-ChildItem *.csv | % { Import-Csv $_ }
start by copying the first file to an output file.
then see this (question, answer):
https://stackoverflow.com/a/2076557/1331076
it shows you how to remove the first line from a file.
you would have to modify it, so instead of replacing your existing file, you Add-Content to the output file.
When dealing with csv files using the cmdlet Import-Csv in PowerShell, is there an easy / elegant way to query a column by its position instead of the header name?
Example with this data:
row1, abc
row2, def
row3, xyz
The below command works fine if the file has "header_column2" as the header of the 2nd column:
import-csv data.csv | where {$_.header_column2 -eq "xyz"}
I am looking for something like this (which obviously doesn't work!):
import-csv data.csv | where {$_.[2] -eq "xyz"}
I know that if I don't have headers in the file I could use the -Header parameter to specify those, but I am looking for something quick and handy to write on the fly for various files, where headers are not relevant.
Many thanks!
Since #PetSerAl doesn't understand the difference between comments and answers here at StackOverflow, I'll provide a similar answer.
You can list the properties in the object using .PSObject.Properties. This returns an IEnumerable that doesn't know about indexes, but if you convert it to an array, it will.
#($_.PSObject.Properties)[2].Value
I've started to play around with PowerShell some time ago, in order to filter some logs one of my servers is creating.
The individual log is a CSV in text file, where first line is some info about the process creating it. Headers are on the 2nd line, and the actual things are on the 3rd. There are about 15 properties, but I only need couple of them.
Here is what works for me flawlessly on one file:
Import-csv file.txt | Select-Object -Skip 1 -Property prop1, prop2, prop3, prop4, prop5 | Export-csv result.csv -NoTypeInformation
But, whatever I tried to use for multiple files (let's say, all .txt files in said folder, since the logs are created per day, and grouped in folders), it doesn't work for me, and I suspect it's because of the different first line, which I try to skip the same way, but I then get empty merged CSV file with only prop1 as 1st column
Any help is appreciated, thanks!
If the headers are actually on the second line, not the first, then you should probably do
Get-Content file.txt | Select-Object -Skip 1 | ConvertFrom-Csv | Export-Csv result.csv -NoTypeInformation
Because this strips the first line before it gets parsed as CSV.
If you want to merge multiple files in the same way, you can do that similarly:
Get-ChildItem *.txt | ForEach-Object {
Get-Content $_ | Select-Object -Skip 1 | ConvertFrom-Csv
} | Export-Csv result.csv -NoTypeInformation