I have a CSV file that contains lines like this:
"Title","Path"
"News","/somepath/news.json/$variable1$variable2"
"Boards","/somepath/$year/$variable3/boardfile.json"
I set all these variables in the script and then import data using Import-Csv cmdlet. As a result, I get object with Title and Path properties and that's what I need. But the problem is that the variables in the CSV file are not getting replaced with actual values during import.
I know there is a solution with Get-Content and ExpandVariable, but it produces an array of strings, and I would rather have an object (or maybe a hashtable?) with properties similar to what Import-Csv produces.
Is there a way to replace variables with values while still using Import-Csv and get a full-featured object as a result?
Import-Csv doesn't expand variables in imported data, so the values in your CSV behave as if they were in single quoted strings. You need something like this to get the variables expanded after import:
Import-Csv 'C:\path\to\your.csv' | Select-Object Title,
#{n='Path';e={$ExecutionContext.InvokeCommand.ExpandString($_.Path)}}
Related
This question already has answers here:
Convert nested JSON array into separate columns in CSV file
(2 answers)
Powershell nested JSON to csv conversion
(4 answers)
Closed 5 months ago.
I have been trying to export multiple data fields from a single json file using powershell, and trying to place them nicely into an excel document. We basically have multiple data fields in each json file that are the same that contain contact information, and we are trying to export a list of that data to place into a new record system.
The flow is as follows: Export ClientID, ContactInfo, CustomerWebsite from each json object (BUT some have multiple entries, its archaic) to an excel CSV that lists File Name | ClientID, ContactInfo, CustomerWebsite.
I have been using the following code block in powershell to do it manually per each field I am trying to export >
Select-String -Path .\*.json -Pattern 'ClientID' | Select-Object -Property Filename,Line | Export-Csv "C:\Users\User\appdev\powershell-scripts\ClientID.csv"
This exports the data in such a fashion:
exported data
Really looking for a way to stream line this into one call, and one singular output file so I can automate this. Ive looked everywhere. I am open to using python instead of powershell, Powershell is just something I know a lot better.
EDIT:
I have tried to use json psobject properties, but an running into issues as well.
Running
$json = (Get-Content "test.json" -Raw) | ConvertFrom-Json
$json.psobject.properties.name
Will only output the first line of the code, which is
sfg_ping::qa::standard_sp_connections
Any sub tag,object listed under that data seemingly isnt accessible by the psobject options. I am not sure if I am doing it incorrectly or what.
fairly new to powershell and I have given myself a bit of a challenge which I think should be possible, I'm just not sure about the best way around it.
We have a user who has a large number of columns in a csv (can vary from 20-50), rows can vary between 1 and 10,000. the data is say ClientName,Address1,Address2,Postcode etc.. (although these can vary wildly depending on the source of the data - external companies) This needs importing into a system using a pre-built routine which looks at the file and needs the database column headers as the csv headers. so say ClientDisplay,Ad_Line1,Ad_Line2,PCode etc..
I was thinking along the lines of either a generic powershell 'mapping' form which could read the headers from ExternalSource.csv and either a DatabaseHeaders.csv (or a direct sql query lookup) display them as columns in a form and then highlight one from each column and a 'link' button, once you have been through all the columns in ExternalSource.csv a 'generate csv' button which takes the mapped headers an appends the correct data columns from ExternalSource.csv
Am I barking up the wrong tree completely trying to use powershell? at the moment its a very time consuming process so just trying to make life easier for users.
Any advice appreciated..
Thanks,
Jon
You can use the Select-Object cmdlet with dynamic columns to shape the data into the form you need.
Something like:
Import-Csv -Path 'source.svc' |
Select-Object Id, Name, #{ Name='Ad_Line1'; Expression={ $_.Address1 } } |
Export-Csv -Path 'target.csv'
In this example, the code #{ Name='Ad_Line1'; Expression={ $_.Address1 } } is a dynamic column, that creates a column with name AD_Line1' and the value ofAddress1`
It is possible to read the column mappings from a file, you will have to write some code to read the file, select the properties and create the format.
A very simple solution could be to read the Select-Object part from another script file, so you can differentiate that part for each import.
A (simple, naive, low performant) solution could look like this (untested code):
# read input file
$input = Import-Csv -Path $inputFile
# read source, target name columns from mapping file
$mappings = Import-Csv -Path $mappingFile | Select Source, Target
# apply transformations
$transformed = $input
foreach($mapping in $mappings) {
# collect the data, add an extra column for each mapping
$transformed = $transformed | Select-Object *, #{ Name = $mapping.Target; Expression = { $_.$($mapping.Source) } }
}
#export transformed data
$transformed | Export-Csv -Path $outputFile
Alternatively; It is possible to convert the data into XML with Import-Csv | Export-CliXml, apply an Xslt template on the Xml to perform a transformation, and save the Xml objects into Csv again with Import-CliXml | Export-Csv.
See this blog by Scott Hansleman on how you can use XSLT with PowerShell.
I've got the following problem:
Whenever I import a .csv file, I can't use the header array elements correctly.
Example:
$data = Import-CSV data.csv
Now, trying to access the data with $data.Surname or whatever, is just empty. Giving out $data in general displays all the information from the csv file. Does anyone know what the problem is here?
I got multiple csv files, all with the same header, all in the same folder which I have to combine to create one large csv file. Since every .csv-file has the same header, I guess using it once should be enough.
All files look like this (also delimited by ','):
Header1,Header2,Header3
Data1, Data2, Data3
Data4, Data5, Data6
Can you help out? I'm not very comfortable with Powershell yet, tried out different codes but nothing really helped me out.
Thanks
if all csv's has the same columns, simply:
# Original CSV
$csv = import-csv c:\temp.csv
# 2nd CSV
$csv2 = import-csv c:\temp2.csv
# As simple as:
$csv += $csv2
If you want to import all CSV's in the current folder you can do something like the following:
Get-ChildItem *.csv | % { Import-Csv $_ }
start by copying the first file to an output file.
then see this (question, answer):
https://stackoverflow.com/a/2076557/1331076
it shows you how to remove the first line from a file.
you would have to modify it, so instead of replacing your existing file, you Add-Content to the output file.
When dealing with csv files using the cmdlet Import-Csv in PowerShell, is there an easy / elegant way to query a column by its position instead of the header name?
Example with this data:
row1, abc
row2, def
row3, xyz
The below command works fine if the file has "header_column2" as the header of the 2nd column:
import-csv data.csv | where {$_.header_column2 -eq "xyz"}
I am looking for something like this (which obviously doesn't work!):
import-csv data.csv | where {$_.[2] -eq "xyz"}
I know that if I don't have headers in the file I could use the -Header parameter to specify those, but I am looking for something quick and handy to write on the fly for various files, where headers are not relevant.
Many thanks!
Since #PetSerAl doesn't understand the difference between comments and answers here at StackOverflow, I'll provide a similar answer.
You can list the properties in the object using .PSObject.Properties. This returns an IEnumerable that doesn't know about indexes, but if you convert it to an array, it will.
#($_.PSObject.Properties)[2].Value