I want to Insert a Blob in a MySQL Database via "MySql.Data.MySqlClient.MySqlConnection", it works fine but Powershell removes the Newline Characters from my Variable:
$configString = Get-Content config.xml
$configString > test5.xml
$strAddConfig = "INSERT INTO config_xml(Version,ConfigXML,MD5,Comment,ClientMinVersion) VALUES('2','" + $($configString) + "','$configMD5','BLABLA','5.0.0.0')"
When I pipe the $configString variable into a Textfile, the CR&LF characters will keep, also when I input the blob via copy paste the Newline characters wil considered. I tried the "strAddConfig" line with different Quote combinations, but I wasn`t successful.
Thanks for help
# Read the text content *as a single, multi-line string*, using -Raw
$configString = Get-Content -Raw config.xml
# Use an expandable here-string for better formatting.
$strAddConfig = #"
INSERT INTO config_xml(Version,ConfigXML,MD5,Comment,ClientMinVersion)
VALUES('2','$configString','$configMD5','BLABLA','5.0.0.0')
"#
Your (primary) problem was the use of Get-Content config.xml without -Raw, which stored an array of lines in $configString, and when an array is used in an expandable string ("...", with string interpolation), its (stringified) elements are space-separated:
PS> $array = 'one', 'two'; "$array"
one two
Related
I am currently writing a script for the following problem:
Initial Problem
Data was exported from an Audit System into a CSV. The CSV itself consists of several columns of which one column has JSON data inside. Sadly there arent many options to influence the export / structure of the export. Since the amount of data included there is tough to filter and to analyse, the exported CSVs (when needed) need to be transformed so that only relevant columns and JSON keys are remaining within the new to-be-exported CSV. It has to be a new CSV as the file needs to potentially be shared. A textfile to-be-imported contains the relevant JSON keys that should remain in the to-be-exported CSV.
About the JSON: The keys can vary based on the events that are exported. Lets say there are 3-4 different variants but the textfile to be imported contains for all 3-4 keys the relevant subkeys that need to be included as new column in the export. If the subkey does not exist its okay if that particular column is empty in the export.
Initial Thoughts
a) Import the CSV and the file that is listing the relevant JSON keys that should be kept
b) Expand the JSON
c) Select the JSON entries that are relevant
d) Merge everything into new columns
e) Export again into a new file
What Questions are open / Where are the Problems?
I was writing some piece of code (I started my PS experience just 2 days ago) and encountered/wondering the following:
Are there any recommendations to improve the code since I am sure to do my very recent PS adventure there are probably many obvious things that have to be improved
Is there a way to make the export straight into a CSV format without the manual -join and then using Out-File?. I noticed that for my final test cases (I cannot share those because the data is extremely hard to anonymize) I didnt manage to come up with a delimiter (tried ",", ";" and "´t") that doesnt seem to be included in parts of the imported cells. Excel (when importing from text) doesnt seem to have an issue tho loading and parsing the data as CSV and to recognize the columns and boundaries correctly.
Happy to hear any tips!
Current Code
### Configure variables
$inputPath = "C:\Users\test\Downloads\inputTest.csv"
$formatTemplate = "C:\Users\test\Downloads\templateTest.txt"
$outputPath = "C:\Users\test\Downloads\outputTest.csv"
### Load the columns from template file to perform transformation depending on the required AuditData Fields. The file contains a list of relevant JSON keys from the Audit Data columns
$selectedAuditDataFields = Get-Content $formatTemplate
### Load CSV, select needed columns and expand the JSON entries within the AuditData column
$importCsvCompact = Import-Csv -Path $inputPath -Delimiter "," | Select-Object -Property CreationDate, UserIds, Operations, #{name = "AuditData"; Expression = {$_.AuditData | ConvertFrom-Json }}
### Calculate the number of Rows (import and export CSV have same number of rows) and Columns (3 standard columns plus template columns) for the CSV to be exported
$exportCsvNumberOfRows = $importCsvCompact.Count
$exportCsvNumberOfColumns = $selectedAuditDataFields.Length + 3
### Add header to to-be-exported-CSV
$header = [object[]]::new($exportCsvNumberOfColumns);
$header[0] = "CreationDate"
$header[1] = "UserIds"
$header[2] = "Operations"
for($columnIncrement = 3; $columnIncrement -ne $exportCsvNumberOfColumns; $columnIncrement++) {
$header[$columnIncrement] = ($selectedAuditDataFields[$columnIncrement-3])
}
$toAppend = $header -join ","
$toAppend | Out-File -FilePath $outputPath -Append
### initiate array for each transformed row and initiate counter of current row
$processingRowCounter = 0
### traverse each row of the CSV import and setup the new column structure
### connect the 3 standard columns with a subset of the expanded JSON entries (based on the imported template)
$importCsvCompact | ForEach-Object {
$csvArrayColumn = [object[]]::new($exportCsvNumberOfColumns);
$csvArrayColumn[0] = $importCsvCompact.CreationDate[$processingRowCounter]
$csvArrayColumn[1] = $importCsvCompact.UserIds[$processingRowCounter]
$csvArrayColumn[2] = $importCsvCompact.Operations[$processingRowCounter]
for($columnIncrement = 3; $columnIncrement -ne $exportCsvNumberOfColumns; $columnIncrement++) {
$csvArrayColumn[$columnIncrement] = $importCsvCompact.AuditData.($selectedAuditDataFields[$columnIncrement-3])[$processingRowCounter]
}
$processingRowCounter++
$directExport = $csvArrayColumn -join ","
$directExport | Out-File -FilePath $outputPath -Append
Write-Host "Processed $processingRowCounter Rows..."
}
Testfiles
templateTest.txt
https://easyupload.io/vx7k75
inputTest.csv
https://easyupload.io/ab77q9
Current Version based on Feedback
### Configure variables
$inputPath = "C:\Users\forstchr\Downloads\inputTest.csv"
$formatTemplate = "C:\Users\forstchr\Downloads\templateTest.txt"
$outputPath = "C:\Users\forstchr\Downloads\outputTest.csv"
### Load the columns from template file to perform transformation depending on the required AuditData Fields. The file contains a list of relevant JSON keys from the Audit Data columns
$selectedAuditDataFields = Get-Content $formatTemplate
### Calculate the number of Rows (import and export CSV have same number of rows) and Columns (3 standard columns plus template columns) for the CSV to be exported
$exportCsvNumberOfRows = $importCsvCompact.Count
$exportCsvNumberOfAuditColumns = $selectedAuditDataFields.Length
###Load CSV, select needed columns and expand the JSON entries within the AuditData column
Import-csv -Path $inputPath -Delimiter "," | Select-Object -Property CreationDate, UserIds, Operations, #{name = "AuditData"; Expression = {$_.AuditData | ConvertFrom-Json }} | % {
[pscustomobject]#{
'CreationDate' = $_.CreationDate
'UserIds' = $_.UserIds
'Operations' = $_.Operations
# the next part is not correct but hopefully displays what I am trying to achieve with the JSON in the AuditData column
for($auditFieldIncrement = 0; $auditFieldIncrement -ne $exportCsvNumberOfAuditColumn; $auditFieldIncrement++) {
'$selectedAuditDataFields[$auditFieldIncrement]' = $_.AuditData.($selectedAuditDataFields[$auditFieldIncrement])
}
}
} | Export-csv $outputPath
I have had to produce a "cleansed" csv file in one project. My general approach was as follows: import the existing csv data, and send it through the pipeline.
Foreach-object, do some processing, storing the results in variables. The last step in processing creates a hashtable typecast as a pscustomobject, and this result in passed through the pipeline. The output of the second pipeline is fed to Export-csv. Export-csv does all the joining and the commas for me, and also encloses the output fields in quotes, making them strings.
Here is a code snippet that illustrates the approach. The cleansing consists of reformatting dates so that they use a standard 14 digit format, and reformatting currency amounts so that they don't contain dollar signs. But that is not relevant to you.
Import-csv checking.csv | % {
$balance += [decimal]$(Get-Amount $_.AMOUNT)
[pscustomobject]#{
'TRNTYPE' = Get-Type $_.AMOUNT
'DTPOSTED' = (Get-Date $_.DATE).Tostring('yyyyMMddHHmmss')
'TRNAMT' = Get-Amount $_.AMOUNT
'FITID' = $fitid++ #this is a stopgap
'NAME' = $_.DESCRIPTION
'MEMO' = $memo
}
} |
Export-csv transactions.csv
Get-Type is a function that yields 'CREDIT' or 'DEBIT' depending on the sign of the amount. Get-Amount is a function that gives a numeric amount without commas and dollar signs. Those functions are defined at the beginning of the script. Note that, when you call a powershell function, there are no parentheses involved. That was a big jolt to me, but it's actually a feature of powershell.
I need to extract Data from a single line of json-data which is inbetween two variables (Powershell)
my Variables:
in front of Data:
DeviceAddresses":[{"Id":
after Data:
,"
I tried this, but there needs to be some error because of all the special characters I'm using:
$devicepattern = {DeviceAddresses":[{"Id":{.*?},"}
#$deviceid = [regex]::match($changeduserdata, $devicepattern).Groups[1].Value
#$deviceid
As you've found, some character literals can't be used as-is in a regex pattern because they carry special meaning - we call these meta-characters.
In order to match the corresponding character literal in an input string, we need to escape it with \ -
to match a literal (, we use the escape sequence \(,
for a literal }, we use \}, and so on...
Fortunately, you don't need to know or remember which ones are meta-characters or escapable sequences - we can use Regex.Escape() to escape all the special character literals in a given pattern string:
$prefix = [regex]::Escape('DeviceAddresses":[{"Id":')
$capture = '(.*?)'
$suffix = [regex]::Escape(',"')
$devicePattern = "${prefix}${capture}${suffix}"
You also don't need to call [regex]::Match directly, PowerShell will populate the automatic $Matches variable with match groups whenever a scalar -match succeeds:
if($changeduserdata -match $devicePattern){
$deviceid = $Matches[1]
} else {
Write-Error 'DeviceID not found'
}
For reference, the following ASCII literals needs to be escaped in .NET's regex grammar:
$ ( ) * + . ? [ \ ^ { |
Additionally, # and (regular space character) needs to be escaped and a number of other whitespace characters have to be translated to their respective escape sequences to make patterns safe for use with the IgnorePatternWhitespace option (this is not applicable to your current scenario):
\u0009 => '\t' # Tab
\u000A => '\n' # Line Feed
\u000C => '\f' # Form Feed
\u000D => '\r' # Carriage Return
... all of which Regex.Escape() takes into account for you :)
To complement Mathias R. Jessen's helpful answer:
Generally, note that JSON data is much easier to work with - and processed more robustly - if you parse it into objects whose properties you can access - see the bottom section.
As for your regex attempt:
Note: The following also applies to all PowerShell-native regex features, such as the -match, -replace, and -split operators, the switch statement, and the Select-String cmdlet.
Mathias' answer uses [regex]::Escape() to escape the parts of the regex pattern to be used verbatim by the regex engine.
This is unequivocally the best approach if those verbatim parts aren't known in advance - e.g., when provided via a variable or expression, or passed as an argument.
However, in a regex pattern that is specified as a string literal it is often easier to individually \-escape the regex metacharacters, i.e. those characters that would otherwise have special meaning to the regex engine.
The list of characters that need escaping is (it can be inferred from the .NET Regular-Expression Quick Reference):
\ ( ) | . * + ? ^ $ [ {
If you enable the IgnorePatternWhiteSpace option (which you can do inline with
(?x), at the start of a pattern), you'll additionally have to \-escape:
#
significant whitespace characters (those you actually want matched) specified verbatim (e.g., ' ', or via string interpolation,"`t"); this does not apply to those specified via escape sequences (e.g., \t or \n).
Therefore, the solution could be simplified to:
# Sample JSON
$changeduserdata = '{"DeviceAddresses":[{"Id": 42,"More": "stuff"}]}'
# Note how [ and { are \-escaped
$deviceId = if ($changeduserdata -match 'DeviceAddresses":\[\{"Id":(.*?),"') {
$Matches[1]
}
Using ConvertFrom-Json to properly parse JSON into objects is both more robust and more convenient, as it allows property access (dot notation) to extract the value of interest:
# Sample JSON
$changeduserdata = '{"DeviceAddresses":[{"Id": 42,"More": "stuff"}]}'
# Convert to an object ([pscustomobject]) and drill down to the property
# of interest; note that the value of .DeviceAddresses is an *array* ([...]).
$deviceId = (ConvertFrom-Json $changeduserdata).DeviceAddresses[0].Id # -> 42
I'm quite new to powershell and just need it for a small task so please excuse my complete and utter ineptitude for the language. I was wondering if it were possible to form a json object based off environment variables and a variable that has already been declared earlier in my script. The variable that was already declared is based off a json config named optionsConfig.json and the contents of that file are here.
{"test1": ["options_size", "options_connection", "options_object"],
"test2":["options_customArgs", "options_noUDP", "options_noName"]}
The purpose of the $Options variable in the code below is to take each element in the list value for the respective test and assume that those elements are environment variables in the system, then find their values and form a dictionary object that will be used in the json.
Here is what I have so far.
# Read the JSON file into a custom object.
$configObj = Get-Content -Raw optionsConfig.json |
ConvertFrom-Json
# Retrieve the environment variables whose
# names are listed in the $env:test property
# as name-value pairs.
Get-Item -Path env:* -Include $configObj.$env:testTool
$Options = Get-Item -Path env:* -Include $configObj.$env:testTool |
% {$hash = #{}} {$hash[$_.Name]=$_.Value} {$hash}
The $Options variable looks like so when converted to json
{
"options_size": "default",
"options_object": "forward open",
"options_connection": "connected"
}
I have a few other environment variable values that I would like to be a part of the json object. Those 3 other environment variables I would like the value of are listed below.
$Env.testTool = "test1"
$Env.RecordName = "Record1"
$Env.Target = "Target1"
How would I construct a powershell statement to get the json object to be formatted like this? -
data = {"test": $Env.testTool, "target": "$Env.Target",
"options": "$Options", "RecordName': "$Env.RecordName"}
The keys are all predefined strings and $Options is the dict object from up above. Am I able to form a Json object like this in powershell and how would it be done? Any help would be appreciated. This appears to be the last step in my struggle with powershell.
Here is what I have done.
$jObj = [ordered]#{test= $Env:testTool}
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
Entries of a hashtable literal (#{ ... } or, in its ordered form, [ordered] #{ ... }) must be separated:
either by newlines (each entry on its own line)
or by ; if placed on the same line.
Thus, the following literals are equivalent:
# Multiline form
#{
test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options
}
# Single-line form; separator is ";"
#{ test= $env:testTool; RecordName= $env:RecordName; Target= $env.Target; options=$Options }
Get-Help about_Hashtables has more information.
$jObj = #{test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options}
$jObj | ConvertTo-Json | Set-Content jsonStuff.json
JsonStuff.json is the new json file for the new json object. This syntax for forming $jObj seems to have done the trick.
I have problem with save string to json file.
$newY = "12313tytk1.xp1`F4i12313211ddsada;"
First I read json file
$a = Get-Content 'settings.json' -raw | ConvertFrom-Json
Then updating field
$a.X.y = $newY
And saving file
$a | ConvertTo-Json -Depth 5 | set-content 'settings.json'
There are some problems:
After save Y in file is wrong:
"12313tytk1.xp1F4i12313211ddsada;"
The special characters are missing: `.
File is wrongly formatted. To much spaces
"<" and ">" are changed to: \u003c and \u003e
How to change it?
Bactick ` is an escape character in Powershell. Single quoted strings ' are string literals, so the contents are not evaluated, escaped or the like. Doulbe quoted strings " are evaluated, so the backtick is interpreted as an escape character. See about_Quoting_Rules for more information.
Consider,
PS C:\> $newY = "12313tytk1.xp1`F4i12313211ddsada;"
PS C:\> $newY # Misses the backtick
12313tytk1.xp1F4i12313211ddsada;
PS C:\> $newY2 = '12313tytk1.xp1`F4i12313211ddsada;'
PS C:\> $newY2 # Contains the backtick
12313tytk1.xp1`F4i12313211ddsada;
I have large CSV files that 0.5-2gb+ files I am trying to import with Powershell.
Data looks like so:
Name, Date, Value
"Joe, John", 2016-08-01, "value"
"Smith, Jane", 2016-08-01, "value"
...
I have this function
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
$reader = new-object System.IO.StreamReader($csv)
while (($line = $reader.ReadLine()) -ne $null) {
# Use RegEx to only split on (,) outside quotes and remove quoted strings
$row = ($line -split ',(?=(?:[^"]|"[^"]*")*$)').Replace("`"","")
# Row Indicator
$i++;
if (($i % 50000) -eq 0) {
Write-Host "$i rows have been processed in $($elapsed.Elapsed.ToString())."
}
}
Splitting the line by a comma "," works perfect as I get ~16K a second, but I need to only split outside of any quotes, so I implemented the regular expression, however the performance tanks to 900 rows a second.
I am looking for a more efficient way to loop through a CSV file that is comma delimited but has commas in the quotes that need to be excluded.
Import-Csv, as noted in the comments above, does not load everything into memory unless you ask it to. Like the example in the question it implements a stream reader and pushes the content it's read off to the output pipeline.
You will see significant memory usage if you do something like this:
$var = Import-Csv thefile.csv
After all, the content of the CSV has to go somewhere.
Whereas if you do something with the output pipeline there's less impact. e.g.
Import-Csv thefile.csv | ForEach-Object {
Do-Something
}
Finally, Import-Csv really doesn't work for you I have a CSV reader class along with a side-by-side implementation of Import-Csv called Indented.Text.Csv on github. This implementation provides a public class with a number of features I needed so I could process CSV files very quickly.