I am trying to insure that only one instance of a perl script can run at one time. The script performs some kind of db_operation depending on the parameters passed in. The script does not necessarily live in one place or on one machine, and possibly multiple OSs. Though the file system is automounted across the various machines.
My first aproach was to just create a .lock file, and do the following:
use warnings;
use strict;
use Fcntl qw(:DEFAULT :flock);
...
open(FILE,">>",$lockFilePath);
flock(FILE,LOCK_EX) or die("Could not lock ");
do_something();
flock(FILE,LOCK_UN) or die("Could not unlock ");
close(FILE);
but I keep getting the following errors:
Bareword "LOCK_EX" not allowed while "strict subs" in use
Bareword "LOCK_UN" not allowed while "strict subs" in use
So I am looking for another way to approach the problem. Locking the DB itself is also not practical since the db could be used by other scripts(which is acceptable), I am just trying to prevent this script from running. And locking a table for write is not practical, since my script is not aware of what table the operation is taking place, it just launches another perl script supplied as a parameter.
I am thinking of adding a table to the db, with just one value, and to use that as a muetex, but I don't know how practical/reliable that is(a lot of red flags go up in my head). I have a DBI connection to a db that this script useses.
Thanks
The Bareword error you are getting sounds like you've done something in that "..." to confuse Perl with regard to the imported Fcntl constants. There's nothing wrong with using those constants like that. You might try something like LOCK_UN() to see what error that gets you.
If you are using MySQL, you can use the GET_LOCK() and RELEASE_LOCK() mechanism. It works reasonably well for cases like this:
SELECT GET_LOCK("script_lock");
and then when you are finished:
SELECT RELEASE_LOCK("script_lock");
See http://dev.mysql.com/doc/refman/4.1/en/miscellaneous-functions.html for details.
You may want to avoid the file locking; from what I remember it's notoriously unreliable on non-local filesystems. Your better bet is to just use the existence of the file itself to the indicator that the script is already running (similar to a UNIX PID file) Granted, this won't be 100% reliable but should work reasonably reliably with very low overhead, provided the script isn't getting invoked incessantly.
If you need better reliability than that, using the database for the mutex is a good solution.
Related
I use sha3 hashes in php (7.2) and I was going to continue using them in MySQL, but MySQL doesn't have support for them yet.
If I write my own UDF and call it, say, sha3, but then some future MySQL version adds a native sha3 function, what happens?
I'll probably name it something else anyway to avoid the possible conflict, but this could happen with any function eventually, so I'm certainly curious.
Just in case anyone is looking for sha3 support in MySQL, I have written a MySQL UDF that does exactly that using rhash that intends to function exactly like the native sha2 function, which you can download here (installation and usage instructions are in the comments) https://gist.github.com/BrianLeishman/a0f40e7a0a87a7069c5c56a768ff3179
Also, it's worth noting that sha3 is supposed to be faster than sha2 (I think), but my function is 4x as slow as native sha2 (when generating 100,000 hashes), but hopefully the future native sha3 will solve that issue.
I've added a separate UDF for returning the hashes without hex encoding called unhex_sha3, which is should literally act as unhex(sha3(..., and this version is almost exactly native speed (compared to sha2), since I can avoid the pointless translation between hex encoding and back.
https://gist.github.com/BrianLeishman/d7903a4acba75707c05fc581e1c714c3
From function name resolution guideline:
Avoid creating UDFs or stored functions that have the same name as a built-in function.
...
If you have already created a user-defined function with a given name and upgrade MySQL to a version that implements a new built-in function with the same name, the UDF becomes inaccessible. To correct this, use DROP FUNCTION to drop the UDF and CREATE FUNCTION to re-create the UDF with a different nonconflicting name. Then modify any affected code to use the new name.
If a new version of MySQL implements a built-in function with the same name as an existing stored function, you have two choices: Rename the stored function to use a nonconflicting name, or change calls to the function so that they use a schema qualifier (that is, use schema_name.func_name() syntax). In either case, modify any affected code accordingly.
someone injects the following content by unsing textfields in forms:
IF(SUBSTR(##version,1,1)<5,BENCHMARK(2600000,SHA1(0xDEADBEEF)),SLEEP(5))/*'XOR(IF(SUBSTR(##version,1,1)<5,BENCHMARK(2600000,SHA1(0xDEADBEEF)),SLEEP(5)))OR'|"XOR(IF(SUBSTR(##version,1,1)<5,BENCHMARK(2600000,SHA1(0xDEADBEEF)),SLEEP(5)))OR"*/
do you have an Idea, what it does? For me, it looks like a try to slow things down.
You can find a lot of injected websites (using this code) in google. I reckon there is some "super hacker script" that is used for that. It seems to use "sample#email.tst" as default email adress. Does somebody know that script?
It's designed to hit the CPU hard regardless of whether the input box's value is unquoted, single-quoted, or double-quoted on older versions of MySQL, and on newer versions, to sleep for 5 seconds, holding the connection open.
In each case it's likely to perform a denial-of-service attack if the application is vulnerable to SQL injection, because holding a connection open for a long time is likely to result in the server running out of resources/available connections.
-- if unquoted, it sees this:
IF(SUBSTR(##version,1,1)<5,BENCHMARK(2600000,SHA1(0xDEADBEEF)),SLEEP(5))
---and then ignores the rest, which appears commented:
/*
-- If it's single-quoted, it doesn't see the comment,
-- rather, it terminates the singlequote:
'
-- ...and then sees this:
XOR(IF(SUBSTR(##version,1,1)<5,BENCHMARK(2600000,SHA1(0xDEADBEEF)),SLEEP(5)))OR
--- ...and then sees the next part as a single-quoted string terinated in the client
'|
--but if it's a double-quoted, string, it sees the end double-quote:
"
-- ...and runs this:
XOR(IF(SUBSTR(##version,1,1)<5,BENCHMARK(2600000,SHA1(0xDEADBEEF)),SLEEP(5)))OR
---and then opens a doublequote to be closed in the client
"
-- This is the end of the comment opened in the case of the unquoted client string.
*/
In each case it's attempting to benchmark an execution of the SHA1 function, which is quite CPU-intensive. BENCHMARK is simply a function that executes another expression a fixed number of times. In this case it's used to perform a CPU DOS on the host.
So I'm creating a vim script that needs to load and parse a JSON file into a local object graph. I searched and I couldn't find any native way to process a JSON file, and I don't want to add any dependencies to the script. So I wrote my own function to parse the JSON string (gotten from the file), but it's really slow. At the moment, I iterate through each character in the file like so:
let len = strlen(jsonString) - 1
let i = 0
while i < len
let c = strpart(jsonString, i, 1)
let i += 1
" A lot of code to process file....
" Note: I've tried short cutting the process by searching for enclosing double-quotes when I come across the initial double quotes (also taking into account escaping '\' character. It doesn't help
endwhile
I've also tried this method:
for c in split(jsonString, '\zs')
" Do a lot of parsing ....
endfor
For reference, a file with ~29,000 characters takes about 4 seconds to process, which is unacceptable.
Is there a better way to iterate over a string in vim script?
Or better yet, have I missed a native function to parse JSON?
Update:
I asked for no dependencies because I:
Didn't want to deal with them
Genuinely wanted some ideas for best way to do this without someone else's work.
Sometimes I just like to do things manually even though the problem has already been solved.
I'm not against plugins or dependencies at all, it's just that I'm curious. Thus the question.
I ended up creating my own function to parse the JSON file. I was creating a script that could parse the package.json file associated with node.js modules. Because of this, I could rely on a fairly consistent format and quit the processing whenever I'd retrieved the information I needed. This usually cut out large chunks of the file since most developers put the largest chunk of the file, their "readme" section, at the end. Because the package.json file is strictly defined, I left the process somewhat fragile. It assumed a root dictionary { } and actively looks for certain entries. You can find the script here: https://github.com/ahayman/vim-nodejs-complete/blob/master/after/ftplugin/javascript.vim#L33.
Of course, this doesn't answer my own question. It's only the solution to my unique problem. I'll wait a few days for new answers and pick the best one before the bounty ends (already set an alarm on my phone).
The simplest solution with the least dependencies is just using the json_decode vim function.
let dict = json_decode(jsonString)
Even though Vim's origin dates back a lot it happens that its internal string() eval() representation is that close to JSON that its likely to work unless you need special characters.
You can lookup the implementation here which even supports true/false/null if you want:
https://github.com/MarcWeber/vim-addon-json-encoding
Better use that library (vim-addon-manager allows to install dependencies easily).
Now it depends on your data whether this is good enough.
Now Benjamin Klein posted your question to vim_use which is why I'm replying.
Best and fast replies happen if you subscribe to the Vim mailinglist.
Goto vim.sf.net and follow the community link.
You cannot expect the Vim community to scrape stackoverflow.
I've added the keyword "json" and "parsing" to that little code that it can be found easier.
If this solution does not work for you you can try the many :h if_* bindings or write an external script which extracts the information you're looking for, or turns JSON into Vim's dictionary representation which can be read by eval() escaping special characters you care about correctly.
If you seek for completely correct solution omitting dependencies is one of the worst thing you can do. The eval() variant mentioned by #MarcWeber is one of the fastest, but it has its disadvantages:
Using solution for securing eval I mentioned in comment makes it no longer the fastest. In fact after you use this it makes eval() slower by more then an order of magnitude (0.02s vs 0.53s in my test).
It does not respect surrogate pairs.
It cannot be used to verify that you have correct JSON: it accepts some strings (e.g. "\<C-o>") that are not JSON strings and it allows trailing commas.
It fails to give normal error messages. It fails badly if you use vam#VerifyIsJSON I mentioned in p.1.
It fails to load floating point values like 1e10 (vim requires numbers to look like 1.0e10, but numbers like 1e10 are allowed: note “and/or” in the first paragraph).
. All of the above (except for the first) statements also apply to vim-addon-json-encoding mentioned by #MarcWeber because it uses eval. There are some other possibilities:
Fastest and the most correct is using python: pyeval('json.loads(vim.eval("varname"))'). Not faster then eval, but fastest among other possibilities. (0.04 in my test: approximately two times slower then eval())
Note that I use pyeval() here. If you want solution for vim version that lacks this functionality it will no longer be one of the fastest.
Use my json.vim plugin. It has an advantages of slightly better error reporting compared to failed vam#VerifyIsJSON, slightly worse compared to eval() and it correctly loads floating-point numbers. It can be used for verification of strings (it does not accept "\<C-a>"), but it loads lists with trailing comma just fine. It does not support surrogate pairs. It is also very slow: in the test I used (it uses 279702 character long strings) it takes 11.59s to load. Json.vim tries to use python if possible though.
For the best error reporting you can take yaml.vim and purge YAML support out of it leaving only JSON (I once have done the same thing for pyyaml, though in python: see markedjson library used in powerline: it is pyyaml minus YAML stuff plus classes with marks). But this variant is even slower then json.vim and should only be used if the main thing you need is error reporting: 207 seconds for loading the same 279702 character long string.
Note that the only variant mentioned that satisfies both requirements “no dependencies” and “no python” is eval(). If you are not fine with its disadvantages you have to throw away one or both of these requirements. Or copy-paste code. Though if you take speed into account only two candidates are left: eval() and python: if you want to parse json fast you really must use C and only these solutions spend most time in functions written in C.
Most other interpreters (ruby/perl/TCL) do not have pyeval() equivalent so they will be slower even if their JSON implementation is written in C. Some other (lua/racket (mzscheme)) have pyeval() equivalent, but e.g. luaeval('{}') is zero meaning that you will have to add additional step explicitly and recursively converting objects into vim dictionaries and lists (e.g. luaeval('vim.dict({})')) which will impact performance. Cannot say anything about mzeval(), but I have never heard about anybody actually using racket (mzscheme) with vim.
I'm creating a set of SQL INSERT statements for a database that doesn't exist yet, and I'm saving them to file.
How can I use Perl's powerful DBI module to create those INSERT statements without accessing a specific database. In particular, it looks like using the $dbh->quote() function requires that I instantiate $dbh with a connection to a database.
Unfortunately, the actual quote() behaviour isn't always a portable characteristic, so each driver will do them differently. Unless you connect to a driver, you don't know which quoting format to use in practice. (There is one module that might do this without a connection, DBIx::Abstract, but it is not especially current.).
The quote() method is actually implemented by the corresponding driver class, in the DBD::* namespace. You might attempt to load the driver you need and call the function directly (see http://search.cpan.org/~timb/DBI-1.616/lib/DBI/DBD.pm#Writing_DBD::Driver::db::quote) but this feels grubby.
I'd still make a DBI connection, if only so that you get the right format of quoting. You don't need to actually send it any statements, but then you do know that the quoting format will be correct for the database you will use.
From DBI::quote:
For most database types, at least those that conform to SQL standards, quote would return 'Don''t' (including the outer quotation marks). For others it may return something like 'Don\'t'
That is, the behavior of DBI::quote varies from database to database, and it doesn't make sense to call it in a database-independent way.
Make a trivial connection to a database of the same type you are writing for, or learn your database's quoting conventions and implement a quote method yourself. See the DBI source for a reference implementation.
Usually you would use DBI by specifying a database like so:
my $dbh = DBI->connect("DBI:mysql:database=$db_name;host=127.0.0.1");
However, your database does not yet exist so you cannot connect to it. You can use DBI without specifying a database like so:
my $dbh = DBI->connect("DBI:mysql:;host=127.0.0.1");
You could use DBD::CSV or DBD::AnyData as a dummy database. SQLite is good for this purpose, too.
A hidden advantage of using SQLite here is that it's a semi-real database, and will tend to make you write code in a way that's decoupled from any specific database.
According to perl -MDBI -E 'say join(q{,},DBI->available_drivers);'
in clean Debian chroot with only DBI (package "libdbi-perl") installed the following drivers are available right away:
DBM,ExampleP,File,Gofer,Proxy,Sponge
The minimum DBI connect statement that works for me is
my $dbh=DBI->connect("DBI:DRIVER:"); # DRIVER is one of [DBM,File,ExampleP,Sponge,mysql,SQLite]
That is enough to use $dbh->quote() with no database whatsoever.
DBM and File escape q{'} as q{\'} ("mysql" style);
ExampleP and Sponge escape: q{'} as q{''} ("SQLite" style).
You can also use:
DBD::_::db->quote()
To access the quote function without setting up a database handle. I don't believe it is specific to MySQL though.
Is it a requirement of good TCL code? What would happen if we don't use the "unset" keyword in a script? Any ill-effects I should know about?
I'm inheriting some legacy code and the errors that come about due to "unset"-ing non-existent variables are driving me up the wall!
It's possible to determine whether a variable exists before using it, using the info exists command. Be sure that if you're not using unset, that you don't upset the logic of the program somewhere else.
There's no Tcl-specific reason to unset a variable, that is, it's not going to cause a memory leak or run out of variable handles or anything crazy like that. Using unset may be a defensive programming practice, because it prevents future use of a variable after it's no longer relevant. Without knowing more about the exact code you're working with, it's hard to give more detailed info.
In addition to the other responses, if your Tcl version is new enough, you can also use:
unset -nocomplain foo
That'll unset foo if it exists, but won't complain if it doesn't.
Depends on the system stats it may give "unable to allocate bytes" issue as and when your script is storing huge data into variables and arrays. it'll break once the cache or RAM is full saying "unable to allocate XXXXXXXX bytes".
Make sure you're not storing that much data into variables, otherwise do unset once the use is over for the respective datasets(variables)
For note as I don't seem able to comment on the "info exists" above;
I use this form often..
if { [info exists pie] && [$pie == "ThisIsWhatIWantInPie"]} {
puts "I found what I wanted in pie."
} else {
puts "Pie did not exist; but I still did not error,TCL's evaluation \
will see the conditional failed on the [info exists] and not \
continue onto the comparison."
}
In addition to the other responses, I want to add that, if you want to neglect the errors raising as a result of unsetting non-existent variable use 'catch'.
#!/bin/bash
catch {unset newVariable}