substitution within a vim function using regex - function

I'm trying to turn a macro into a Function. Whilst I can save the macro I'd like to make it a function I can call.
I'm de-duping a specific element in a large concatenation of xml files using the following
G$b"myw0"n4ywdd:264501?^Rn/\dcw^Rm^[
Which gets parts of the last line into registers m and n then goes to a known line above the dedupe data and searches backwards for the contents of register n and replaces the number in the line with register m. The known line is to make sure I find the first occurrence that's not part of the de-dedupe set. Works fine if a bit slow when running 600 times.
On the function I've managed to duplicate everything except the cw with the contents of register m part.
function! MakeUnique()
let i = 0
while i <= 1
:200
:let #n=""
:let #m=""
:normal! G
:normal! $
:normal! b
:normal! "myw
:normal! 0
:normal! "n4yw
:200
execute "call search('".#n."','b')"
let i += 1
endwhile
endfunction
I've tried a few combinations of substitutions along the lines of the following but with no success.
Try 1
Register n = <dataid>389 and register m = <dataid>333
I changed the Line at the end of the file for this try to and changed "nyw to ny$
<dataId>333<dataid> <dataId>389
I would expect that subcommand when executed would read . s/<dataId>333/<dataId>389/
let subCommand=". s/".#m"/'".#n."'/"
exec subCommand
But given the result it doesn't seem as if #n is expanding
Result:
</dataid>
Try 2
In this case #n = 389
let subCommand=". s/[0-9][0-9][0-9]/".#n."/"
exec subCommand
This one works but the \d+ part may b 3 - 5 digits and I can't get a regex to work in this case, ie
s/\(\d\+\)/".#m."/
doesn't work and I get a E486: Pattern not found d+
Questions:
Is there any way to use regex match in the let subcommand = bit eg let subCommand = ". s/\d+/".#n"/"
Alternatively how do I replace using something like
let subCommand=". s/".#m"/'".#n."'/"
Or is there a vimscript equivalent of cw^Rm ?
Data example, cut down to fit.
The first iteration of the macro will change 6415 to 8790 in the identified line below.
I'm trying to replicate that in a Function.
1 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
2 <createsaleRequest>
3 <sale>
4 <saleNumber>9987166</saleNumber>
5 <dataId>6315</dataId>
6 </sale>
7 ...
8 </createsaleRequest>
9
10
11 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
12 <createsaleRequest>
13 <sale>
14 <saleNumber>9988166</saleNumber>
15 <dataId>6415</dataId>
16 </sale>
17 ...
18 </createsaleRequest>
19
20 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
21 <createsaleRequest>
22 <sale>
23 <saleNumber>9987921</saleNumber>
24 <dataId>6315</dataId>
25 </sale>
26 ...
27 </createsaleRequest>
28
29
30 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
31 <createsaleRequest>
32 <sale>
33 <saleNumber>9987411</saleNumber>
34 <dataId>6415</dataId> <!-- Change this on first run through -->
35 </sale>
36 ...
37 </createsaleRequest>
38
39 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
40 <createsaleRequest>
41 <sale>
42 <saleNumber>9987411</saleNumber>
43 <dataId>6193</dataId>
44 </sale>
45 ...
46 </createsaleRequest>
47
48
49
50
51 <dataId>6315</dataId> 6789
52 <dataId>6415</dataId> 8790
EDIT
Added a cut down data example to try and make the question make sense.
Edit
Found the problem. I'd left the second . out of the match portion. This let subCommand=". s/".#m."/".#n."/"works and is significantly faster than the macro. Don't know how to close the question.
I would like to know what ". ." means. I'm guessing it's an escape pattern that works in vimscript.

Related

not accepting conversion out of range mysql go decimal

I am writing a Go program that communicates with MySQL.
I have the following column:
lot_size double(8,8)
I'm ingesting an xml feed and parsing it in Go. Here is the data dump of the lot_size xml data
(string) (len=4) "0.36"
Also, sometimes it can be:
(string) ""
In my Go program, I have the following conversion:
if listings.Listings[i].LotSize == "" {
lotSize=0.00000000
} else {
lotSize,_=strconv.ParseFloat(listings.Listings[i].LotSize,8)
}
Mysql is producing the following error:
Error 1264: Out of range value for column 'lot_size' at row 1
Not sure what to do at this point
UPDATE
HERE is the parsed float data output:
(float64) 0
34 (float64) 0.410239
35 (float64) 0.045914
36 (float64) 0.325941
37 (float64) 0.208425
38 (float64) 0.078627
39 (float64) 0.132989
40 (float64) 0.163522
41 (float64) 0.24
42 (float64) 0.230119
43 (float64) 0
44 (float64) 4.25
my sql query is:
insert into listings (lot_size) values (?)
stmt,err:=db.Prepare(sqlStatement)
res,err:=stmt.Exec(lotSize)
The answer discussed in comments above boils down to this:
MySQL FLOAT(8,8) type means the value may have eight digits, but all eight must be to the right of the decimal. This means any value 1.0 or greater cannot be stored in this column if you declare a FLOAT with two argument that are the same number.
This behavior is documented here: https://dev.mysql.com/doc/refman/8.0/en/floating-point-types.html
By the way, this nonstandard form of FLOAT with two arguments is deprecated in MySQL 8.0.17 and will be removed in a future version of MySQL. It's recommended to use DECIMAL instead.

Decoding a hex file

I would like to use a webservice who deliver a binary file with some data, I know the result but I don't know how I can decode this binary with a script.
Here is my binary :
https://pastebin.com/3vnM8CVk
0a39 0a06 3939 3831 3438 1206 4467 616d
6178 1a0b 6361 7264 6963 6f6e 5f33 3222
0d54 6865 204f 6c64 2047 7561 7264 2a02
....
Some part are in ASCII so it easy to catch, at the end of the file you got vehicle name in ASCII and some data, it should be kill/victory/battle/XP/Money data but I don't understand how I can decode these hexa value, I tried to compare 2 vehicles who got same kills but I don't see any match.
There is a way to decode these data ?
Thanks :)
Hello guys, after 1 year I started again to find a solution, so here is the structure of the packet I guessed : (the part between [ ] I still don't know what is it for)
[52 37 08 01 10] 4E [18] EA [01 25] AB AA AA 3E [28] D4 [01 30] EC [01 38] 88 01 [40] 91 05 [48] 9F CA 22 [50] F5 C2 9A 02 [5A 12]
| | | | | | | | |
Victories Victory Ratio| | Air target| Xp Money earned
| | | Ground Target
Battles Deaths Respawns
So here is the result :
Victory : 78
Battles : 234
Victory Ratio : ? (should be arround 33%)
Deaths : 212
Respawns : 236
Air Target : 136
Ground Target : 657
Xp : ? (should be arround 566.56k)
Money : ? (should be arround 4.63M)
Is there a special way to calculate the result of a long hex like this ?
F5 C2 9A 02 (should be arround 4.63M)
I tell you a bit more :
I know the result, but I don't know how to calculate it with these hex from the packet.
If I check a packet with a small amout of money or XP to be compatible with one hex :
[52 1E 08 01 10] 01 [18] [01 25] 00 00 80 3F [28] 01 [30] 01 [48] 24 [50] 6E [5A 09]
6E = 110 Money earned
24 = 36 XP earned
Another exemple :
[52 21 08 01 10] 02 [18] 03 [25] AB AA 2A 3F [28] 02 [30] 03 [40] 01 [48] 78 [50] C7 08 [5A 09]
XP earned = hex 78 = 120
Money earned = hex C7 08 = 705
How C7 08 can do 705 decimal ?
Here is the full content in case but I know how to isolate just these part I don't need to decode all these hex data :
https://pastebin.com/vAKPynNb
What you have asked is nothing but how to reverse engineer a binary file. Lot of threads already on SO
Reverse engineer a binary dictionary file to extract strings
Tools to help reverse engineer binary file formats
https://reverseengineering.stackexchange.com/questions/3495/what-tools-exist-for-excavating-data-structures-from-flat-binary-files
http://www.iwriteiam.nl/Ha_HTCABFF.html
The final take out on all is that no single solution for you, you need to spend effort to figure it out. There are tools to help you, but don't expect a magic wand tool to give you the structure/data.
Any kind of file read operation is done in text or binary format with basic file handlers. And some languages offer type reading of int, float etc. or arrays of them.
The complex operations behind these reading are almost always kept hidden from normal users. And then the user has to learn more when it comes to read/write operations of data structures.
In this case, OFFSET and SEEK are the words one must find value and act accordingly. whence data read, it must be converted to suitable data type too.
The following code shows basics for these operations to write data and read blocks to get numbers back. It is written in PHP as the OP has commented in the question he uses PHP.
Offset is calculated with these byte values to be 11: char: 1 byte, short: 2 bytes, int: 4 bytes, float: 4 bytes.
<?php
$filename = "testdata.dat";
$filehandle = fopen($filename, "w+");
$data=["test string","another test string",77,777,77777,7.77];
fwrite($filehandle,$data[0]);
fwrite($filehandle,$data[1]);
$numbers=array_slice($data,2);
fwrite($filehandle,pack("c1s1i1f1",...$numbers));
fwrite($filehandle,"end"); // gives 3 to offset
fclose($filehandle);
$filename = "testdata.dat";
$filehandle = fopen($filename, "rb+");
$offset=filesize($filename)-11-3;
fseek($filehandle,$offset);
$numberblock= fread($filehandle,11);
$numbersback=unpack("c1a/s1b/i1c/f1d",$numberblock);
var_dump($numbersback);
fclose($filehandle);
?>
Once this example understood, the rest is to find the data structure in the requested file. I had written another example but it uses assumptions. I leave the rest to readers to find what assumptions I made here. Be careful though: I know nothing about real structure and values will not be correct.
<?php
$filename = "testfile";
$filehandle = fopen($filename, "rb");
$offset=17827-2*41; //filesize minus 2 user area
fseek($filehandle,$offset);
print $user1 = fread($filehandle, 41);echo "<br>";
$user1pr=unpack("s1kill/s1victory/s1battle/s1XP/s1Money/f1Life",$user1);
var_dump($user1pr); echo "<br>";
fseek($filehandle,$offset+41);
print $user2 = fread($filehandle, 41);echo "<br>";
$user2pr=unpack("s1kill/s1victory/s1battle/i1XP/i1Money/f1Life",$user2);
var_dump($user2pr); echo "<br>";
echo "<br><br>";
$repackeduser2=pack("s3i2f1",$user2pr["kill"],$user2pr["victory"],
$user2pr["battle"],$user2pr["XP"],$user2pr["Money"],
$user2pr["Life"]
);
print $user2 . "<br>" .$repackeduser2;
print "<br>3*s1=6bytes, 2*i=6bytes, 1*f=*bytes (machine dependent)<br>";
print pack("s1",$user2pr["kill"]) ."<br>";
print pack("s1",$user2pr["victory"]) ."<br>";
print pack("s1",$user2pr["battle"]) ."<br>";
print pack("i1",$user2pr["XP"]) ."<br>";
print pack("i1",$user2pr["Money"]) ."<br>";
print pack("f1",$user2pr["Life"]) ."<br>";
fclose($filehandle);
?>
PS: pack and unpack uses machine dependent size for some data types such as int and float, so be careful with working them. Read Official PHP:pack and PHP:unpack manuals.
This looks more like the hexdump of a binary file. Some methods of converting hex to strings resulted in the same scrambled output. Only some lines are readable like this...
Dgamaxcardicon_32" The Old Guard
As #Tarun Lalwani said, you would have to know the structure of this data to get the in plaintext.
If you have access to the raw binary, you could try using strings https://serverfault.com/questions/51477/linux-command-to-find-strings-in-binary-or-non-ascii-file

rjson::fromJSON returns only the first item

I have a sqlite database file with several columns. One of the columns has a JSON dictionary (with two keys) embedded in it. I want to extract the JSON column to a data frame in R that shows each key in a separate column.
I tried rjson::fromJSON, but it reads only the first item. Is there a trick that I'm missing?
Here's an example that mimics my problem:
> eg <- as.vector(c("{\"3x\": 20, \"6y\": 23}", "{\"3x\": 60, \"6y\": 50}"))
> fromJSON(eg)
$3x
[1] 20
$6y
[1] 23
The desired output is something like:
# a data frame for both variables
3x 6y
1 20 23
2 60 50
or,
# a data frame for each variable
3x
1 20
2 60
6y
1 23
2 50
What you are looking for is actually a combination of lapply and some application of rbind or related.
I'll extend your data a little, just to have more than 2 elements.
eg <- c("{\"3x\": 20, \"6y\": 23}",
"{\"3x\": 60, \"6y\": 50}",
"{\"3x\": 99, \"6y\": 72}")
library(jsonlite)
Using base R, we can do
do.call(rbind.data.frame, lapply(eg, fromJSON))
# X3x X6y
# 1 20 23
# 2 60 50
# 3 99 72
You might be tempted to do something like Reduce(rbind, lapply(eg, fromJSON)), but the notable difference is that in the Reduce model, rbind is called "N-1" times, where "N" is the number of elements in eg; this results in a LOT of copying of data, and though it might work alright with small "N", it scales horribly. With the do.call option, rbind is called exactly once.
Notice that the column labels have been R-ized, since data.frame column names should not start with numbers. (It is possible, but generally discouraged.)
If you're confident that all substrings will have exactly the same elements, then you may be good here. If there's a chance that there will be a difference at some point, perhaps
eg <- c(eg, "{\"3x\": 99}")
then you'll notice that the base R solution no longer works by default.
do.call(rbind.data.frame, lapply(eg, fromJSON))
# Error in (function (..., deparse.level = 1, make.row.names = TRUE, stringsAsFactors = default.stringsAsFactors()) :
# numbers of columns of arguments do not match
There may be techniques to try to normalize the elements such that you can be assured of matches. However, if you're not averse to a tidyverse package:
library(dplyr)
eg2 <- bind_rows(lapply(eg, fromJSON))
eg2
# # A tibble: 4 × 2
# `3x` `6y`
# <int> <int>
# 1 20 23
# 2 60 50
# 3 99 72
# 4 99 NA
though you cannot call it as directly with the dollar-method, you can still use [[ or backticks.
eg2$3x
# Error: unexpected numeric constant in "eg2$3"
eg2[["3x"]]
# [1] 20 60 99 99
eg2$`3x`
# [1] 20 60 99 99

Have Browserify transform Hintify continue on error

I'm using Gulp with Browserify and Hintify. The standard way to catch errors seems to be something like:
browserify({
entries: 'app.js',
transform: [
// Some other transforms
plugins.hintify
]
// A bunch of other settings
}).bundle()
// Error handling
.on('error', function(error) {
util.beep(); // Util is gulp-util
util.log(error);
this.emit('end');
})
.pipe(...)
This works great, and everytime I violate my jshint I get something like this:
Users/me/Sites/project/client/scripts/app.js: line 11, col 23, Missing semicolon.
And the build stops.
However, I'd kinda like the build to continue even if I forget a semicolon. Right now, any jshint error stops the build from processing. I like to check if my code is up to par, but it shouldn't prevent everything from building if I have an unused argument in a function.
Help?
The current setup works for me (configuration sam as here):
var jshint = require('gulp-jshint');
var notify = require('gulp-notify');
var stylish = require('jshint-stylish')
...
.pipe(jshint('.jshintrc'))
.pipe(jshint.reporter('jshint-stylish'))
//.pipe(jshint.reporter('fail')) // <- omit a fail reporter
...
OR
...
.pipe(jshint('.jshintrc'))
.pipe(jshint.reporter('jshint-stylish'))
.pipe(jshint.reporter('fail'))
.on('error', notify.onError({ message: 'JS hint fail'})) // <- pipe fail to notification
...
This gives me the output:
...
src\m\typings.js
line 0 col 0 ES5 option is now set per default
line 3 col 18 ['SUCCESS'] is better written in dot notation.
line 4 col 18 ['FAILURE'] is better written in dot notation.
line 5 col 18 ['UNSTABLE'] is better written in dot notation.
line 6 col 18 ['NOT_BUILT'] is better written in dot notation.
line 10 col 26 ['info'] is better written in dot notation.
line 11 col 26 ['success'] is better written in dot notation.
line 12 col 26 ['mark'] is better written in dot notation.
line 13 col 26 ['warn'] is better written in dot notation.
line 14 col 26 ['error'] is better written in dot notation.
line 15 col 26 ['debug'] is better written in dot notation.
line 16 col 26 ['none'] is better written in dot notation.
line 18 col 1 Unnecessary semicolon.
‼ 13 warnings
[10:08:32] src\m\typings.js
src\helper\scaffolding.helper.js
line 0 col 0 ES5 option is now set per default
line 65 col 61 Regular parameters cannot come after default parameters.
line 82 col 79 Expected an assignment or function call and instead saw an expression.
line 84 col 77 Regular parameters cannot come after default parameters.
line 90 col 39 Expected an assignment or function call and instead saw an expression.
line 91 col 21 Missing 'new' prefix when invoking a constructor.
line 91 col 60 Expected an assignment or function call and instead saw an expression.
line 105 col 46 Expected an assignment or function call and instead saw an expression.
line 106 col 29 Missing 'new' prefix when invoking a constructor.
line 106 col 103 Expected an assignment or function call and instead saw an expression.
line 3 col 12 'require' is not defined.
line 4 col 14 'require' is not defined.
line 5 col 11 'require' is not defined.
line 6 col 19 'require' is not defined.
line 7 col 10 'require' is not defined.
line 8 col 13 'require' is not defined.
line 9 col 13 'require' is not defined.
line 33 col 17 'none' is not defined.
line 42 col 13 'color' is not defined.
line 45 col 13 'color' is not defined.
line 48 col 13 'color' is not defined.
line 51 col 13 'color' is not defined.
line 54 col 13 'color' is not defined.
line 57 col 13 'color' is not defined.
line 60 col 13 'color' is not defined.
line 63 col 12 'color' is not defined.
line 85 col 25 'process' is not defined.
× 2 errors
‼ 25 warnings
[10:08:32] src\helper\scaffolding.helper.js
src\helper\scaffolding.server.js
line 0 col 0 ES5 option is now set per default
line 22 col 44 Expected an assignment or function call and instead saw an expression.
line 23 col 25 Missing 'new' prefix when invoking a constructor.
line 23 col 75 Expected an assignment or function call and instead saw an expression.
line 25 col 43 Expected an assignment or function call and instead saw an expression.
line 26 col 25 Missing 'new' prefix when invoking a constructor.
line 26 col 117 Expected an assignment or function call and instead saw an expression.
line 38 col 40 Expected an assignment or function call and instead saw an expression.
line 39 col 21 Missing 'new' prefix when invoking a constructor.
line 39 col 50 Expected an assignment or function call and instead saw an expression.
line 74 col 43 Expected an assignment or function call and instead saw an expression.
line 75 col 25 Missing 'new' prefix when invoking a constructor.
line 75 col 96 Expected an assignment or function call and instead saw an expression.
line 4 col 12 'require' is not defined.
line 5 col 13 'require' is not defined.
line 7 col 13 'require' is not defined.
‼ 16 warnings
[10:08:32] src\helper\scaffolding.server.js
src\server\publish.js
line 0 col 0 ES5 option is now set per default
line 17 col 17 Missing 'new' prefix when invoking a constructor.
line 17 col 54 Expected an assignment or function call and instead saw an expression.
line 29 col 21 Missing 'new' prefix when invoking a constructor.
line 29 col 44 Expected an assignment or function call and instead saw an expression.
line 3 col 12 'require' is not defined.
line 4 col 13 'require' is not defined.
line 7 col 14 'require' is not defined.
line 9 col 12 'require' is not defined.
line 10 col 10 'require' is not defined.
line 11 col 14 'require' is not defined.
line 12 col 13 'require' is not defined.
‼ 12 warnings
[10:08:32] gulp-notify: [Error running Gulp] JS hint fail
[10:08:32] src\server\publish.js
[10:08:32] Finished 'build' after 1.61 s
with jshint ^1.12.0

Strip_Tags yielding strange result and how to display HTML tags as text

Good morning!
I'm trying to scrape a CME web page to pull the table at the bottom to a two dimensional array. (http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html)
Code is below. Problem is, the var_dump says string(507) but there are only about 300 characters displayed! Three questions:
1) How do I display any hidden tags or characters?
2) Why does it say 507 chars but only about 300 chars displayed?
3) How do I remove whatever characters are hidden?
Thank you for your help!
Here is the code I used:
$EMiniURL = "http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html";
$EMiniRaw = file_get_contents($EMiniURL);
$EMiniRaw = strip_tags($EMiniRaw);
$StartChr = strpos($EMiniRaw, "About This Report") + strlen("About This Report");
$EndChr = strpos($EMiniRaw, "Total", $StartChr) - strlen("Total");
$TotalLen = $EndChr - $StartChr;
$RawStr = substr ($EMiniRaw, $StartChr, $TotalLen);
var_dump ($RawStr);
And here is the var_dump result:
string(532) " DEC 14 1938.50 1964.50 1935.75 1959.75 +21.75 1960.25 1,551,405 2,751,445 MAR 15 1931.00 1956.25B 1928.00A 1952.00A +21.75 1952.50 2,244 5,495 JUN 15 1920.25 1949.00B 1920.25 1945.00A +22.00 1945.50 88 350 SEP 15 1925.00 1937.75B 1925.00 1937.75B +21.75 1938.75 6 204 DEC 15 1935.75 1935.75 1935.75 1935.75 +22.00 1932.75 1 212 "
Turns out there were newlines and tabs. Apparently PHP strip_tags does not remove everything. Used the following to finish the clean up:
$NewStr = preg_replace("/\s+/", " ", $OldStr);
Just saying ;)