Powershell array value in MySQL insert is outputting entire array - mysql
First off I am new to Powershell, as in this is my first ever attempt. I'm doing the I can program in other languages so I can hack through this project. I have spent a few days now trying to solve this one, and I know is something stupid. I'm just stuck.
# Insert into master table
$query = "INSERT INTO ``master``(``diff``, ``key``) VALUES ('$diff', '$key')"
Invoke-MySqlQuery -Query $query
$query
This works fine the test output displays:
INSERT INTO `master`(`diff`, `key`) VALUES ('248', 'k000002143200000000000000680006080005500900030082670009461000500000091000000000000')
It also inputs into the MySQL DB just fine.
This following the exact same format is not working for me. Like I said at the top, I know it is some stupid formatting thing I'm missing.
# Insert into answers table
$query = "INSERT INTO ``puzzles``(``id``, ``p1``, ``p2``, ``p3``, ``p4``, ``p5``, ``p6``, ``p7``, ``p8``, ``p9``, ``p10``, ``p11``, ``p12``, ``p13``, ``p14``, ``p15``, ``p16``, ``p17``, ``p18``, ``p19``, ``p20``, ``p21``, ``p22``, ``p23``, ``p24``, ``p25``, ``p26``, ``p27``, ``p28``, ``p29``, ``p30``, ``p31``, ``p32``, ``p33``, ``p34``, ``p35``, ``p36``, ``p37``, ``p38``, ``p39``, ``p40``, ``p41``, ``p42``, ``p43``, ``p44``, ``p45``, ``p46``, ``p47``, ``p48``, ``p49``, ``p50``, ``p51``, ``p52``, ``p53``, ``p54``, ``p55``, ``p56``, ``p57``, ``p58``, ``p59``, ``p60``, ``p61``, ``p62``, ``p63``, ``p64``, ``p65``, ``p66``, ``p67``, ``p68``, ``p69``, ``p70``, ``p71``, ``p72``, ``p73``, ``p74``, ``p75``, ``p76``, ``p77``, ``p78``, ``p79``, ``p80``, ``p81``) VALUES ('$id', '$p[0]', '$p[1]', '$p[2]', '$p[3]', '$p[4]', '$p[5]', '$p[6]', '$p[7]', '$p[8]', '$p[9]', '$p[10]', '$p[11]', '$p[12]', '$p[13]', '$p[14]', '$p[15]', '$p[16]', '$p[17]', '$p[18]', '$p[19]', '$p[20]', '$p[21]', '$p[22]', '$p[23]', '$p[24]', '$p[25]', '$p[26]', '$p[27]', '$p[28]', '$p[29]', '$p[30]', '$p[31]', '$p[32]', '$p[33]', '$p[34]', '$p[35]', '$p[36]', '$p[37]', '$p[38]', '$p[39]', '$p[40]', '$p[41]', '$p[42]', '$p[43]', '$p[44]', '$p[45]', '$p[46]', '$p[47]', '$p[48]', '$p[49]', '$p[50]', '$p[51]', '$p[52]', '$p[53]', '$p[54]', '$p[55]', '$p[56]', '$p[57]', '$p[58]', '$p[59]', '$p[60]', '$p[61]', '$p[62]', '$p[63]', '$p[64]', '$p[65]', '$p[66]', '$p[67]', '$p[68]', '$p[69]', '$p[70]', '$p[71]', '$p[72]', '$p[73]', '$p[74]', '$p[75]', '$p[76]', '$p[77]', '$p[78]', '$p[79]', '$p[80]')"
$query
$p[0]
$p[1]
$p[2]
$p[3]
$p[4]
$p[5]
Invoke-MySqlQuery -Query $query
This is the output I get on one of the runs:
INSERT INTO `puzzles`(`id`, `p1`, `p2`, `p3`, `p4`, `p5`, `p6`, `p7`, `p8`, `p9`, `p10`, `p11`, `p12`, `p13`, `p14`, `p15`, `p16`, `p17`, `p
18`, `p19`, `p20`, `p21`, `p22`, `p23`, `p24`, `p25`, `p26`, `p27`, `p28`, `p29`, `p30`, `p31`, `p32`, `p33`, `p34`, `p35`, `p36`, `p37`, `p
38`, `p39`, `p40`, `p41`, `p42`, `p43`, `p44`, `p45`, `p46`, `p47`, `p48`, `p49`, `p50`, `p51`, `p52`, `p53`, `p54`, `p55`, `p56`, `p57`, `p
58`, `p59`, `p60`, `p61`, `p62`, `p63`, `p64`, `p65`, `p66`, `p67`, `p68`, `p69`, `p70`, `p71`, `p72`, `p73`, `p74`, `p75`, `p76`, `p77`, `p
78`, `p79`, `p80`, `p81`) VALUES ('2596', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[0]', '000002143
200000000000000680006080005500900030082670009461000500000091000000000000[1]', '0000021432000000000000006800060800055009000300826700094610005
00000091000000000000[2]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[3]', '0000021432000000000000006
80006080005500900030082670009461000500000091000000000000[4]', '00000214320000000000000068000608000550090003008267000946100050000009100000000
0000[5]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[6]', '00000214320000000000000068000608000550090
0030082670009461000500000091000000000000[7]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[8]', '00000
2143200000000000000680006080005500900030082670009461000500000091000000000000[9]', '000002143200000000000000680006080005500900030082670009461
000500000091000000000000[10]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[11]', '0000021432000000000
00000680006080005500900030082670009461000500000091000000000000[12]', '0000021432000000000000006800060800055009000300826700094610005000000910
00000000000[13]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[14]', '00000214320000000000000068000608
0005500900030082670009461000500000091000000000000[15]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[1
6]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[17]', '000002143200000000000000680006080005500900030
082670009461000500000091000000000000[18]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[19]', '0000021
43200000000000000680006080005500900030082670009461000500000091000000000000[20]', '0000021432000000000000006800060800055009000300826700094610
00500000091000000000000[21]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[22]', '00000214320000000000
0000680006080005500900030082670009461000500000091000000000000[23]', '00000214320000000000000068000608000550090003008267000946100050000009100
0000000000[24]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[25]', '000002143200000000000000680006080
005500900030082670009461000500000091000000000000[26]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[27
]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[28]', '0000021432000000000000006800060800055009000300
82670009461000500000091000000000000[29]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[30]', '00000214
3200000000000000680006080005500900030082670009461000500000091000000000000[31]', '00000214320000000000000068000608000550090003008267000946100
0500000091000000000000[32]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[33]', '000002143200000000000
000680006080005500900030082670009461000500000091000000000000[34]', '000002143200000000000000680006080005500900030082670009461000500000091000
000000000[35]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[36]', '0000021432000000000000006800060800
05500900030082670009461000500000091000000000000[37]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[38]
', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[39]', '00000214320000000000000068000608000550090003008
2670009461000500000091000000000000[40]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[41]', '000002143
200000000000000680006080005500900030082670009461000500000091000000000000[42]', '000002143200000000000000680006080005500900030082670009461000
500000091000000000000[43]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[44]', '0000021432000000000000
00680006080005500900030082670009461000500000091000000000000[45]', '0000021432000000000000006800060800055009000300826700094610005000000910000
00000000[46]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[47]', '00000214320000000000000068000608000
5500900030082670009461000500000091000000000000[48]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[49]'
, '000002143200000000000000680006080005500900030082670009461000500000091000000000000[50]', '000002143200000000000000680006080005500900030082
670009461000500000091000000000000[51]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[52]', '0000021432
00000000000000680006080005500900030082670009461000500000091000000000000[53]', '0000021432000000000000006800060800055009000300826700094610005
00000091000000000000[54]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[55]', '00000214320000000000000
0680006080005500900030082670009461000500000091000000000000[56]', '00000214320000000000000068000608000550090003008267000946100050000009100000
0000000[57]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[58]', '000002143200000000000000680006080005
500900030082670009461000500000091000000000000[59]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[60]',
'000002143200000000000000680006080005500900030082670009461000500000091000000000000[61]', '0000021432000000000000006800060800055009000300826
70009461000500000091000000000000[62]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[63]', '00000214320
0000000000000680006080005500900030082670009461000500000091000000000000[64]', '00000214320000000000000068000608000550090003008267000946100050
0000091000000000000[65]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[66]', '000002143200000000000000
680006080005500900030082670009461000500000091000000000000[67]', '000002143200000000000000680006080005500900030082670009461000500000091000000
000000[68]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[69]', '0000021432000000000000006800060800055
00900030082670009461000500000091000000000000[70]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[71]',
'000002143200000000000000680006080005500900030082670009461000500000091000000000000[72]', '00000214320000000000000068000608000550090003008267
0009461000500000091000000000000[73]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[74]', '000002143200
000000000000680006080005500900030082670009461000500000091000000000000[75]', '000002143200000000000000680006080005500900030082670009461000500
000091000000000000[76]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[77]', '0000021432000000000000006
80006080005500900030082670009461000500000091000000000000[78]', '0000021432000000000000006800060800055009000300826700094610005000000910000000
00000[79]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[80]')
0
0
0
0
0
2
In the $query variable the $p[X] does not work like the $p[X] below. It's spitting out the entire array. Obviously it's not working when it sends it to MySQL, I get the ID correct and that's it.
Thank you all for the help in advance, thumping my head on the keyboard hurts after awhile!
Ian
Bonus question! Any tips on this would be great as well. Any of the 0s don't need to be there, I actually want them as NULL. Any tips on a clean way to do that would be wonderful. Full disclosure I have not even looked into this part yet!
EDIT - In response to #Mathias R. Jessen
I removed all the other code not having to do with $p, to make it easy on the eyes.
# Set the key variable
$key = (Get-Content Puzzles.txt)[$loop] # ....73.146...8.......6......5..2..8.8..5.9..1.923..........273.........8..3..1.46 #Extreme
$key = $key.Replace(".", "0") # 000073014600080000000600000050020080800509001092300000000002730000000008003001046 #Extreme
$key = -join('k', $key.Substring(0,81)) # k000073014600080000000600000050020080800509001092300000000002730000000008003001046
$check = (Invoke-MySqlQuery -Query "SELECT COUNT(*) FROM ``master`` WHERE ``key`` = '$key'")
$check = $check.'COUNT(*)'
# Check in master table if key exists, i.e. not a new puzzle
IF ($check -eq 0) {
$p = $key.Substring(1,81) # 000073014600080000000600000050020080800509001092300000000002730000000008003001046
# Insert into answers table
$query = "INSERT INTO ``puzzles``(``id``, ``p1``, ``p2``, ``p3``, ``p4``, ``p5``, ``p6``, ``p7``, ``p8``, ``p9``, ``p10``, ``p11``, ``p12``, ``p13``, ``p14``, ``p15``, ``p16``, ``p17``, ``p18``, ``p19``, ``p20``, ``p21``, ``p22``, ``p23``, ``p24``, ``p25``, ``p26``, ``p27``, ``p28``, ``p29``, ``p30``, ``p31``, ``p32``, ``p33``, ``p34``, ``p35``, ``p36``, ``p37``, ``p38``, ``p39``, ``p40``, ``p41``, ``p42``, ``p43``, ``p44``, ``p45``, ``p46``, ``p47``, ``p48``, ``p49``, ``p50``, ``p51``, ``p52``, ``p53``, ``p54``, ``p55``, ``p56``, ``p57``, ``p58``, ``p59``, ``p60``, ``p61``, ``p62``, ``p63``, ``p64``, ``p65``, ``p66``, ``p67``, ``p68``, ``p69``, ``p70``, ``p71``, ``p72``, ``p73``, ``p74``, ``p75``, ``p76``, ``p77``, ``p78``, ``p79``, ``p80``, ``p81``) VALUES ('$id', '$p[0]', '$p[1]', '$p[2]', '$p[3]', '$p[4]', '$p[5]', '$p[6]', '$p[7]', '$p[8]', '$p[9]', '$p[10]', '$p[11]', '$p[12]', '$p[13]', '$p[14]', '$p[15]', '$p[16]', '$p[17]', '$p[18]', '$p[19]', '$p[20]', '$p[21]', '$p[22]', '$p[23]', '$p[24]', '$p[25]', '$p[26]', '$p[27]', '$p[28]', '$p[29]', '$p[30]', '$p[31]', '$p[32]', '$p[33]', '$p[34]', '$p[35]', '$p[36]', '$p[37]', '$p[38]', '$p[39]', '$p[40]', '$p[41]', '$p[42]', '$p[43]', '$p[44]', '$p[45]', '$p[46]', '$p[47]', '$p[48]', '$p[49]', '$p[50]', '$p[51]', '$p[52]', '$p[53]', '$p[54]', '$p[55]', '$p[56]', '$p[57]', '$p[58]', '$p[59]', '$p[60]', '$p[61]', '$p[62]', '$p[63]', '$p[64]', '$p[65]', '$p[66]', '$p[67]', '$p[68]', '$p[69]', '$p[70]', '$p[71]', '$p[72]', '$p[73]', '$p[74]', '$p[75]', '$p[76]', '$p[77]', '$p[78]', '$p[79]', '$p[80]')"
Invoke-MySqlQuery -Query $query
}
You need to use the subexpression operator $() to evaluate the collection at that index during string interpolation. Without this operator, the content of your entire collection is being printed, as well as your literal index syntax.
Your first example works as expected because you're only interpolating simple variables, without doing any additional work to them.
Here's a simple example from the command line:
C:\> $arr = 1,2,3,4
Outside of a string:
C:\> $arr[0]
1
During string interpolation, without the subexpression operator:
C:\> "$arr[0]"
1 2 3 4[0]
During string interpolation, with the subexpression operator:
C:\> "$($arr[0])"
1
This means that your example would become something like this:
...VALUES ('$id', '$($p[0])', '$($p[1])'...
Note that $id is working correctly because it is a simple variable. You only need to use the subexpression operator for additional work like evaluating indexes, properties, etc.
This concept is also sometimes called variable expansion, if you would like to research further.
Related
Cannot index into null array from function returned variable, or issues accessing regex data returned
I'm not sure if I'm returning the value from the function incorrectly, but when I try to access it's info, it has the above error,
Cannot index into a null array
I've tried a couple different ways, and I'm not sure if I'm not returning this correctly from the function, or if I'm just accessing the info returned incorrectly. Looking at Cannot index into null array, it looks like for him, some of his array had null values. But when I print my info to screen before I exit the function, it has info. How do I return the value found in the function such that I can loop through the contents in my main code and use one of the strings in the object? This is a continuation of parsing repeated pattern.
#parse data out of cpp code and loop through to further process
#function
Function Get-CaseContents{
[cmdletbinding()]
Param ( [string]$parsedCaseMethod, [string]$parseLinesGroupIndicator)
Process
{
# construct regex
$fullregex = [regex]"_stprintf[\s\S]*?_T\D*", # Start of error message, capture until digits
"(?<sdkErr>\d+)", # Error number, digits only
"\D[\s\S]*?", # match anything, non-greedy
"(?<sdkDesc>\((.+?)\))", # Error description, anything within parentheses, non-greedy
"([\s\S]*?outError\s*=(?<sdkOutErr>\s[a-zA-Z_]*))", # Capture OutErr string and parse out part after underscore later
"[\s\S]*?", # match anything, non-greedy
"(?<sdkSeverity>outSeverity\s*=\s[a-zA-Z_]*)", # Capture severity string and parse out part after underscore later
'' -join ''
# run the regex
$Values = $parsedCaseMethod | Select-String -Pattern $fullregex -AllMatches
# Convert Name-Value pairs to object properties
$result = foreach ($match in $Values.Matches){
[PSCustomObject][ordered]#{
sdkErr = $match.Groups['sdkErr']
sdkDesc = $match.Groups['sdkDesc']
sdkOutErr = $match.Groups['sdkOutErr']
sdkSeverity = ($match.Groups['sdkSeverity'] -split '_')[-1] #take part after _
}
}
#Write-Host "result:" $result -ForegroundColor Green
$result
return $Values
...
#main code
...
#call method to get case info (sdkErr, sdkDesc, sdkOutErr, sdkSeverity)
$ValuesCase = Get-CaseContents -parsedCaseMethod $matchFound -parseLinesGroupIndicator "_stprintf" #need to get returned info back
$result = foreach ($match in $ValuesCase.Matches){
[PSCustomObject][ordered]#{
sdkErr = $match.Groups['sdkErr']
sdkDesc = $match.Groups['sdkDesc']
sdkOutErr = $match.Groups['sdkOutErr']
sdkSeverity = ($match.Groups['sdkSeverity'] -split '_')[-1] #take part after _
} #result
} #foreach ValuesCase
The example of string sent to the function to parse is:
...
case kRESULT_STATUS_Undefined_Opcode:
_stprintf( outDevStr, _T("8004 - (Comm. Err 04) - %s(Undefined Opcode)"), errorStr);
outError = INVALID_PARAM;
outSeverity = CCA_WARNING;
break;
case kRESULT_STATUS_Comm_Timeout:
_stprintf( outDevStr, _T("8005 - (Comm. Err 05) - %s(Timeout sending command)"), errorStr);
outError = INVALID_PARAM;
outSeverity = CCA_WARNING;
break;
case kRESULT_STATUS_TXD_Failed:
_stprintf( outDevStr, _T("8006 - (Comm. Err 06) - %s(TXD Failed--Send buffer overflow.)"), errorStr);
outError = INVALID_PARAM;
outSeverity = CCA_WARNING;
break;
...
Another thing I tried is (but it also had the index into null array issue):
foreach($matchRegex in $ValuesCase.Matches)
{
$sdkOutErr = $matchRegex.Groups['sdkOutErr']
Write-Host sdkOutErr -ForegroundColor DarkMagenta
}
Ultimately, I need to grab $sdkOutErr to further process. I'll need to use the other variables too in the returned object, but this is the first one I need. I love the way the output is formatted in the function, but probably don't know how to return the info and use what is returned. I'm not sure what to search for to figure out the issue other than the error message, which leads me to believe I'm returning the info wrong. I don't think I need to return $result, because I think that's just a string with the values in the $values.Matches in the function. I need to access the values returned as I mentioned.
I checked, and the contents sent to the function is not blank.
I tried returning $results, and it looks like this when I write-Host, which would be difficult to access each sdkOutErr:
#{sdkErr=1000; sdkDesc=(Out of Memory); sdkOutErr= NO_MEMORY; sdkSeverity=FATAL} #{sdkErr=1002; sdkDesc=(Failed to load DLL); sdkOutErr= OTHER_ERROR; sdkSeverity=FATAL} #{sdkErr=1003; sdkDesc=(Failed to load DLL); sdk
OutErr= OTHER_ERROR; sdkSeverity=FATAL} #{sdkErr=1004; sdkDesc=(Failed to open); sdkOutErr= OTHER_ERROR; sdkSeverity=FATAL} #{sdkErr=1005; sdkDesc=(Unable to access the specified profile); sdkOutErr= OTHER_ERROR; sdkSeverity=
FATAL} #{sdkErr=100 ...
How can I return this from the function so that it's not a null array/index, and the data is accessible if I use a foreach loop (or two) in the main code to get the sdkOutErr (to start).
I'm fairly new to (complicated)powershell and I have a feeling I need a map inside the array in my function, but I'm not sure.
Before I returned the function Values or results, it was printing something like this out. Once I added in main $ValuesCase=Get-CaseContents... (returning $values from function), or $parsedCase = Get-CaseContents... (returning $results from function), it stopped showing this on the screen:
sdkErr sdkDesc sdkOutErr sdkSeverity
------ ------- --------- -----------
1000 (Out of Memory) NO_MEMORY FATAL
1002 (Failed to load DLL) OTHER_ERROR FATAL
1003 (Failed to load DLL) OTHER_ERROR FATAL
1004 (Failed to open) OTHER_ERROR FATAL
I tried returning $results, and it looks like this when I write-Host, which would be difficult to access each sdkOutErr: Getting all the sdkOutErr values is not as difficult as you might imagine: $results.sdkOutErr # this will output the `sdkOutErr` value from each object in the array Or, outside the function: (Get-CaseContents -parsedCaseMethod $matchFound -parseLinesGroupIndicator "_stprintf").sdkOutErr Another option, which might perform better if the result set is large, is to use ForEach-Object to grab just the sdkOutErr values: $fullResults = Get-CaseContents -parsedCaseMethod $matchFound -parseLinesGroupIndicator "_stprintf" $sdkOutErrValuesOnly = $fullResults |ForEach-Object -MemberName sdkOutErr
A shorter non-repeating alphanumeric code than UUID in MySQL
Is it possible for MySQL database to generate a 5 or 6 digit code comprised of only numbers and letters when I insert a record? If so how? Just like goo.gl, bit.ly and jsfiddle do it. For exaple: http://bit.ly/3PKQcJ http://jsfiddle.net/XzKvP cZ6ahF, 3t5mM, xGNPN, xswUdS... So UUID_SHORT() will not work because it returns a value like 23043966240817183 Requirements: Must be unique (non-repeating) Can be but not required to be based off of primary key integer value Must scale (grow by one character when all possible combinations have been used) Must look random. (item 1234 cannot be BCDE while item 1235 be BCDF) Must be generated on insert. Would greatly appreciate code examples.
Try this: SELECT LEFT(UUID(), 6);
I recommend using Redis for this task, actually. It has all the features that make this task suitable for its use. Foremost, it is very good at searching a big list for a value.
We will create two lists, buffered_ids, and used_ids. A cronjob will run every 5 minutes (or whatever interval you like), which will check the length of buffered_ids and keep it above, say, 5000 in length. When you need to use an id, pop it from buffered_ids and add it to used_ids.
Redis has sets, which are unique items in a collection. Think of it as a hash where the keys are unique and all the values are "true".
Your cronjob, in bash:
log(){ local x=$1 n=2 l=-1;if [ "$2" != "" ];then n=$x;x=$2;fi;while((x));do let l+=1 x/=n;done;echo $l; }
scale=`redis-cli SCARD used_ids`
scale=`log 16 $scale`
scale=$[ scale + 6]
while [ `redis-cli SCARD buffered_ids` -lt 5000 ]; do
uuid=`cat /dev/urandom | tr -cd "[:alnum:]" | head -c ${1:-$scale}`
if [ `redis-cli SISMEMBER used_ids $uuid` == 1]; then
continue
fi
redis-cli SADD buffered_ids $uuid
done
To grab the next uid for use in your application (in pseudocode because you did not specify a language)
$uid = redis('SPOP buffered_ids');
redis('SADD used_ids ' . $uid);
edit actually there's a race condition there. To safely pop a value, add it to used_ids first, then remove it from buffered_ids.
$uid = redis('SRANDMEMBER buffered_ids');
redis('SADD used_ids ' . $uid);
redis('SREM buffered_ids ' . $uid);
Inserting multiple rows in a column in MySQL using a Perl program
I want to insert multiple rows of data in a single column using a single query. This program is for less data. I have another weather monitoring .txt file which had 4000 lines of data. I can insert one data at a time but it becomes tedious for so many data values.
1. use DBI;
2. use DBD::mysql;
3. use warnings;
4. $connection = ConnectToMySql($database);
5. # Multiple Data inputs
6. $myquery = "INSERT INTO data(datatime,battery)
7. VALUES
8. (?,?),
9. ('16.01.2013','6.54'), #data corresponding to date and battery
10. ('17.01.2013','6.42'),
11. ('21.01.2013','6.24'),
12. ('22.01.2013','6.21'),
13. ('24.01.2013','6.17'),
14. ('25.01.2013','6.13'),
15. ('28.01.2013','6.00'),
16. ('29.01.2013','5.97'),
17. ('30.01.2013','5.94'),
18. ('01.02.2013','5.84')";
19. $statement2 = $connection->prepare($myquery);
20. $statement2->execute($myquery);
21. #--- start sub-routine
22. sub ConnectToMySql {
23. $database ="xxxx";
24. $user = "XXXX";
25. $pass = "XXXX";
26. $host="XXXX";
27. my $dbh = DBI->connect("DBI:mysql:$database:$host", $user, $pass);
28. }
This code is giving me the following errors:
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 2 at C:/Users/User/workspace/DataBaseEntry/DataEntry.pl line 20.
DBD::mysql::st execute failed: called with 1 bind variables when 2 are needed at C:/Users/User/workspace/DataBaseEntry/DataEntry.pl line 40.
I cannot identify the problem. Is it the placeholder. What can i do to improve it?
I am new to these things. so can you keep it simple.
THANKS
You should be passing the data values which should replace the (?, ?) as parameters to execute. Your code as written only passes a single parameter to execute and that parameter is the SQL text of your query.
Try this instead:
$myquery = "INSERT INTO data(datatime,battery) VALUES (?,?)";
my $sth = $connection->prepare($myquery);
$sth->execute('16.01.2013','6.54');
$sth->execute('17.01.2013','6.42');
$sth->execute('21.01.2013','6.24');
$sth->execute('22.01.2013','6.21');
$sth->execute('24.01.2013','6.17');
$sth->execute('25.01.2013','6.13');
$sth->execute('28.01.2013','6.00');
$sth->execute('29.01.2013','5.97');
$sth->execute('30.01.2013','5.94');
$sth->execute('01.02.2013','5.84');
$connection->do(<<'EOT');
INSERT INTO data (datatime, battery)
VALUES
('17.01.2013', '6.42'),
('21.01.2013', '6.24'),
('22.01.2013', '6.21'),
('24.01.2013', '6.17'),
('25.01.2013', '6.13'),
('28.01.2013', '6.00'),
('29.01.2013', '5.97'),
('30.01.2013', '5.94'),
('01.02.2013', '5.84')
EOT
I'm not sure what you're trying to do with placeholders here.
Also, you're missing use warnings; use strict;, and you shouldn't use global variables everywhere.
Dynamic Multi Insert with DBI placeholders for many sets of VALUES
I'm building a dynamic SQL statement, that will insert one or more sets of VALUES via a prepared DBI statement, my question is this: Since I have a dynamic number of VALUES sets, and I will need to add as many ( ?, ?, ?),( ?, ?, ?) etc as necessary to extend the statement INSERT INTO `tblname` ( $columnsString ) VALUES in order to submit only one query using placeholders and bind values- is this the preferred method(most efficient, etc., - reasoning behind efficiency would be helpful in your answer if possible) or should I just be building this as a query string with sprintf and dbh->quote()? (As a little extra information: I'm actually using AnyEvent::DBI right now, which only exposes placeholders & bind values and not the quote() method so this wouldn't be easy for me to accomplish without creating another straight DBI $dbh and using another db server connection just to use the quote() method, or without altering the AnyEvent::DBI module myself.) Normally I would just execute the statements as necessary but in this heavy workload case I'm trying to batch inserts together for some DB efficiency. Also, if anyone could answer if it is possible( and then how to ) insert an sql DEFAULT value using placeholders and bind values that'd be awesome. Typically if I ever needed to do that I'd append the DEFAULTs to the string directly and use sprintf and $dbh->quote() only for the non DEFAULT values. UPDATE: Worked out the misunderstanding in a quick chat. User ikegami suggested that instead of building the query string myself without placeholders, that I just intermingle VALUES and placeholders such as: $queryString .= '(DEFAULT,?,?),(DEFAULT,DEFAULT,DEFAULT)'; Some of the reasoning behind my first asking of this question on SO was because I was somewhat against this intermingling due to my thought that it made the code less readable, though after being assured that sql 'DEFAULT' couldn't be in a placeholder bind value, this was the method I had begun implementing. Using placeholders where possible does seem to be the more accepted method of building queries, and if you want an SQL DEFAULT you just need to include it in the same query building as the placeholders. This does not apply to NULL values, as those CAN be inserted with placeholders and a bind value of undef. Update 2: The reasoning I asked about performance, the 'acceptance' of building your own query with quote() vs building with placeholders, and why I've gone with a solution that involves using all columns for the SQL INSERT INTO tblname (cols) is because I have roughly 2-4 million rows a day going into a terrible db server, and my code is running on an equally terrible server. With my requirements of needing DEFAULT sql values, and these terrible performance constraints, I've chosen a solution for now. For future devs who stumble upon this - take a look at #emazep's solution of using SQL::Abstract, or if for some reason you need to build your own, you might consider either using #Schwern's subroutine solution or possibly incorporating some of #ikegami's answer into it as these are all great answers as to the 'Current state of affairs' regarding the usage of DBI and building dynamic queries.
Unless there is a specific reason to reinvent the wheel (there could be some), SQL::Abstract (among others) has already solved the problem of dynamic SQL generation for all of us:
my %data = (
name => 'Jimbo Bobson',
phone => '123-456-7890',
address => '42 Sister Lane',
city => 'St. Louis',
state => 'Louisiana'
);
use SQL::Abstract;
my ($stmt, #bind)
= SQL::Abstract->new->insert('people', \%data);
print $stmt, "\n";
print join ', ', #bind;
which prints:
INSERT INTO people ( address, city, name, phone, state)
VALUES ( ?, ?, ?, ?, ? )
42 Sister Lane, St. Louis, Jimbo Bobson, 123-456-7890, Louisiana
SQL::Abstract then offers a nice trick to iterate over many rows to insert without regenerating the SQL every time, but for bulk inserts there is also SQL::Abstract::Plugin::InsertMulti
use SQL::Abstract;
use SQL::Abstract::Plugin::InsertMulti;
my ($stmt, #bind)
= SQL::Abstract->new->insert_multi( 'people', [
{ name => 'foo', age => 23 },
{ name => 'bar', age => 40 },
]);
# INSERT INTO people ( age, name ) VALUES ( ?, ? ), ( ?, ? )
# 23, foo, 40, bar
I have, on occasion, used a construct like:
#!/usr/bin/env perl
use strict; use warnings;
# ...
my #columns = ('a' .. 'z');
my $sql = sprintf(q{INSERT INTO sometable (%s) VALUES (%s)},
join(',', map $dbh->quote($_), #columns),
join(',', ('?') x #columns),
);
As for handling DEFAULT, wouldn't leaving that column out ensure that the DB sets it to the default value?
If you would use placeholders for "static" queries, you should use them for "dynamic" queries too. A query is a query.
my $stmt = 'UPDATE Widget SET foo=?'
my #params = $foo;
if ($set_far) {
$stmt .= ', far=?';
push #params, $far;
}
{
my #where;
if ($check_boo) {
push #where, 'boo=?';
push #params, $boo;
}
if ($check_bar) {
push #where, 'bar=?';
push #params, $bar;
}
$stmt .= ' WHERE ' . join ' AND ', map "($_)", #where
if #where;
}
$dbh->do($stmt, undef, #params);
I used an UPDATE since it allowed me to demonstrate more, but everything applies to INSERT too.
my #fields = ('foo');
my #params = ($foo);
if ($set_far) {
push #fields, 'bar';
push #params, $far;
}
$stmt = 'INSERT INTO Widget ('
. join(',', #fields)
. ') VALUES ('
. join(',', ('?')x#fields)
. ')';
$dbh->do($stmt, undef, #params);
You've expressed concerns about the readability of the code and also being able to pass in a DEFAULT. I'll take #ikegami's answer one step further...
sub insert {
my($dbh, $table, $fields, $values) = #_;
my $q_table = $dbh->quote($table);
my #q_fields = map { $dbh->quote($_) } #$fields;
my #placeholders = map { "?" } #q_fields;
my $sql = qq{
INSERT INTO $q_table
( #{[ join(', ', #q_fields) ]} )
VALUES ( #{[ join(', ', #placeholders ]} )
};
return $dbh->do($sql, undef, #$values);
}
Now you have a generic multi value insert routine.
# INSERT INTO foo ('bar', 'baz') VALUES ( 23, 42 )
insert( $dbh, "foo", ['bar', 'baz'], [23, 43] );
To indicate a default value, don't pass in that column.
# INSERT INTO foo ('bar') VALUES ( 23 )
# 'baz' will use its default
insert( $dbh, "foo", ['bar'], [23] );
You can optimize this to make your subroutine do multiple inserts with one subroutine call and one prepared statement saving CPU on the client side (and maybe some on the database side if it supports prepared handles).
sub insert {
my($dbh, $table, $fields, #rows) = #_;
my $q_table = $dbh->quote($table);
my #q_fields = map { $dbh->quote($_) } #$fields;
my #placeholders = map { "?" } #q_fields;
my $sql = qq{
INSERT INTO $q_table
( #{[ join(', ', #q_fields) ]} )
VALUES ( #{[ join(', ', #placeholders ]} )
};
my $sth = $dbh->prepare_cached($sql);
for my $values (#rows) {
$sth->execute(#$values);
}
}
# INSERT INTO foo ('bar', 'baz') VALUES ( 23, 42 )
# INSERT INTO foo ('bar', 'baz') VALUES ( 99, 12 )
insert( $dbh, "foo", ['bar', 'baz'], [23, 43], [99, 12] );
Finally, you can write a bulk insert passing in multiple values in a single statement. This is probably the most efficient way to do large groups of inserts. This is where having a fixed set of columns and passing in a DEFAULT marker comes in handy. I've employed the idiom where values passed as scalar references are treated as raw SQL values. Now you have the flexibility to pass in whatever you like.
sub insert {
my($dbh, $table, $fields, #rows) = #_;
my $q_table = $dbh->quote($table);
my #q_fields = map { $dbh->quote($_) } #$fields;
my $sql = qq{
INSERT INTO $q_table
( #{[ join(', ', #q_fields) ]} )
VALUES
};
# This would be more elegant building an array and then joining it together
# on ",\n", but that would double the memory usage and there might be
# a lot of values.
for my $values (#rows) {
$sql .= "( ";
# Scalar refs are treated as bare SQL.
$sql .= join ", ", map { ref $value ? $$_ : $dbh->quote($_) } #$values;
$sql .= "),\n";
}
$sql =~ s{,\n$}{};
return $dbh->do($sql);
}
# INSERT INTO foo ('bar', 'baz') VALUES ( 23, NOW ), ( DEFAULT, 12 )
insert( $dbh, "foo", ['bar', 'baz'], [23, \"NOW"], [\"DEFAULT", 12] );
The down side is this builds a string in memory, possibly very large. To get around that you have to involve database specific bulk insert from file syntax.
Rather than writing all this SQL generation stuff yourself, go with #emazep's answer and use SQL::Abstract and SQL::Abstract::Plugin::InsertMulti.
Just make sure you profile.
How do I cleanly extract MySQL enum values in Perl?
I have some code which needs to ensure some data is in a mysql enum prior to insertion in the database. The cleanest way I've found of doing this is the following code:
sub enum_values {
my ( $self, $schema, $table, $column ) = #_;
# don't eval to let the error bubble up
my $columns = $schema->storage->dbh->selectrow_hashref(
"SHOW COLUMNS FROM `$table` like ?",
{},
$column
);
unless ($columns) {
X::Internal::Database::UnknownColumn->throw(
column => $column,
table => $table,
);
}
my $type = $columns->{Type} or X::Panic->throw(
details => "Could not determine type for $table.$column",
);
unless ( $type =~ /\Aenum\((.*)\)\z/ ) {
X::Internal::Database::IncorrectTypeForColumn->throw(
type_wanted => 'enum',
type_found => $type,
);
}
$type = $1;
require Text::CSV_XS;
my $csv = Text::CSV_XS->new;
$csv->parse($type) or X::Panic->throw(
details => "Could not parse enum CSV data: ".$csv->error_input,
);
return map { /\A'(.*)'\z/; $1 }$csv->fields;
}
We're using DBIx::Class. Surely there is a better way of accomplishing this? (Note that the $table variable is coming from our code, not from any external source. Thus, no security issue).
No need to be so heroic. Using a reasonably modern version of DBD::mysql, the hash returned by DBI's column info method contains a pre-split version of the valid enum values in the key mysql_values:
my $sth = $dbh->column_info(undef, undef, 'mytable', '%');
foreach my $col_info ($sth->fetchrow_hashref)
{
if($col_info->{'TYPE_NAME'} eq 'ENUM')
{
# The mysql_values key contains a reference to an array of valid enum values
print "Valid enum values for $col_info->{'COLUMN_NAME'}: ",
join(', ', #{$col_info->{'mysql_values'}}), "\n";
}
...
}
I'd say using Text::CSV_XS may be an overkill, unless you have weird things like commas in enums (a bad idea anyway if you ask me). I'd probably use this instead. my #fields = $type =~ / ' ([^']+) ' (?:,|\z) /msgx; Other than that, I don't think there are shortcuts.
I spent part of the day asking the #dbix-class channel over on MagNet the same question and came across this lack of answer. Since I found the answer and nobody else seems to have done so yet, I'll paste the transcript below the TL;DR here:
my $cfg = new Config::Simple( $rc_file );
my $mysql = $cfg->get_block('mysql');
my $dsn =
"DBI:mysql:database=$mysql->{database};".
"host=$mysql->{hostname};port=$mysql->{port}";
my $schema =
DTSS::CDN::Schema->connect( $dsn, $mysql->{user}, $mysql->{password} );
my $valid_enum_values =
$schema->source('Cdnurl')->column_info('scheme')->{extra}->{list};
And now the IRC log of me beating my head against a wall:
14:40 < cj> is there a cross-platform way to get the valid values of an
enum?
15:11 < cj> it looks like I could add 'InflateColumn::Object::Enum' to the
__PACKAGE__->load_components(...) list for tables with enum
columns
15:12 < cj> and then call values() on the enum column
15:13 < cj> but how do I get dbic-dump to add
'InflateColumn::Object::Enum' to
__PACKAGE__->load_components(...) for only tables with enum
columns?
15:20 < cj> I guess I could just add it for all tables, since I'm doing
the same for InflateColumn::DateTime
15:39 < cj> hurm... is there a way to get a column without making a
request to the db?
15:40 < cj> I know that we store in the DTSS::CDN::Schema::Result::Cdnurl
class all of the information that I need to know about the
scheme column before any request is issued
15:42 <#ilmari> cj: for Pg and mysql Schema::Loader will add the list of
valid values to the ->{extra}->{list} column attribute
15:43 <#ilmari> cj: if you're using some other database that has enums,
patches welcome :)
15:43 <#ilmari> or even just a link to the documentation on how to extract
the values
15:43 <#ilmari> and a willingness to test if it's not a database I have
access to
15:43 < cj> thanks, but I'm using mysql. if I were using sqlite for this
project, I'd probably oblige :-)
15:44 <#ilmari> cj: to add components to only some tables, use
result_components_map
15:44 < cj> and is there a way to get at those attributes without making a
query?
15:45 < cj> can we do $schema->resultset('Cdnurl') without having it issue
a query, for instance?
15:45 <#ilmari> $result_source->column_info('colname')->{extra}->{list}
15:45 < cj> and $result_source is $schema->resultset('Cdnurl') ?
15:45 <#ilmari> dbic never issues a query until you start retrieving the
results
15:45 < cj> oh, nice.
15:46 <#ilmari> $schema->source('Cdnurl')
15:46 <#ilmari> the result source is where the result set gets the results
from when they are needed
15:47 <#ilmari> names have meanings :)