mariadb c-connector bind / execute mess up memory allocation - mysql

i'm using mariadb c-connector with prepare, bind and execute. it works usualy. but one case end up in "corrupted unsorted chunks" and core dumping when freeing bind buffer. i suggest the whole malloc organisation is messed up after calling mysql_stmt_execute(). my test's MysqlDynamic.c show:
the problem only is connected to x509cert variable bound by bnd[9]
freeing memory only fails if bnd[9].is_null = 0, if is_null execute end normally
freeing memory (using FreeStmt()) after bind and before execute end normally
print of bnd[9].buffer before execute show (void*) is connected to the correct string buffer
same behavior for setting bnd[9].buffer_length to STMT_INDICATOR_NTS or strlen()
other similar bindings (picture, bnd[10]) do not lead to corrupted memory and core dump.
i defined a c structure test for test data in my test program MysqlDynamic.c which is bound in MYSQL_BIND structure.
bindings for x509cert (string buffer) see bindInsTest():
bnd[9].buffer_type = MYSQL_TYPE_STRING;
bnd[9].buffer_length = STMT_INDICATOR_NTS;
bnd[9].is_null = &para->x509certI;
bnd[9].buffer = (void*) para->x509cert;
please get the details out of source file MysqlDynamic.c. please adapt defines in the source to your environment, verify content, and run it. you will find compile info in source code. MysqlDynymic -c will create the table. MysqlDynamic -i will insert 3 records each run. And 'MysqlDynamic -d` drop the the table again.
MysqlDynamic -vc show:
session set autocommit to <0>
connection id: 175
mariadb server ver:<100408>, client ver:<100408>
connected on localhost to db test by testA
>> if program get stuck - table is locked
table t_test created
mysql connection closed
pgm ended normaly
MysqlDynamic -i show
ins2: BufPara <92> name<master> stamp<> epoch<1651313806000>
cert is cert<(nil)> buf<(nil)> null<1>
picure is pic<0x5596a0f0c220> buf<0x5596a0f0c220> null<0> length<172>
ins1: BufPara <91> name<> stamp<2020-04-30> epoch<1650707701123>
cert is cert<0x5596a0f181d0> buf<0x5596a0f181d0> null<0>
picure is pic<(nil)> buf<(nil)> null<1> length<0>
ins0: BufPara <90> name<gugus> stamp<1988-10-12T18:43:36> epoch<922337203685477580>
cert is cert<(nil)> buf<(nil)> null<1>
picure is pic<(nil)> buf<(nil)> null<1> length<0>
free(): corrupted unsorted chunks
Aborted (core dumped)
checking t_test table content show all records are inserted as expected.
you can disable loading of x509cert and/or picture by commenting out the defines line 57/58. the program than end normally. you also can comment out line 208. the buffers are then indicated as NULL.
Questions:
is there a generic coding mistake in the program causing this behavior?
can you run the program in your environment without core dumping? i'm currently using version 10.04.08.
any improvment in code will be welcome.

Related

Debezium reads binlog newest if offset storage is empty or just created&

I am running reading binlogs with Debezium, but when I start new reading thread it reads all create statements for table from the beginning, but I dont need them(op=c). I need to handle create/update/delete events that happens after I run code first time. And than work with correct offset (that stored in file "tmp/offsets.dat"), so how I can set initial configuration in this way? So the flow need to be the next one:
start reading(first time) -> take current(latest position from binlog and save it, work from here) and handle newest events
start reading(not the first run) -> take latest position from file and read data as usual
Here is my current configurations
config = Configuration.empty().withSystemProperties(Function.identity()).edit()
.with(MySqlConnectorConfig.SERVER_NAME, SERVER_NAME)
.with(MySqlConnectorConfig.SKIPPED_OPERATIONS, "r")
.with(MySqlConnectorConfig.HOSTNAME, HOSTNAME)
.with(MySqlConnectorConfig.PORT, PORT)
.with(MySqlConnectorConfig.USER, USER)
.with(MySqlConnectorConfig.PASSWORD, PASSWORD)
.with(MySqlConnectorConfig.TABLE_WHITELIST, TABLE_WHITELIST)
.with(MySqlConnectorConfig.SERVER_ID, 100)
//
.with(EmbeddedEngine.OFFSET_STORAGE, "org.apache.kafka.connect.storage.FileOffsetBackingStore")
.with(EmbeddedEngine.OFFSET_STORAGE_FILE_FILENAME, "tmp/offsets.dat")
.with(EmbeddedEngine.CONNECTOR_CLASS, "io.debezium.connector.mysql.MySqlConnector")
.with(EmbeddedEngine.ENGINE_NAME, SERVER_NAME)
//
.with(MySqlConnectorConfig.DATABASE_HISTORY, "io.debezium.relational.history.FileDatabaseHistory")
.with("database.history.file.filename", "tmp/dbhistory.dat")
// Send JSON without schema
.with("schemas.enable", false)
.build();
and my.cnf values for binlogs
[mysqld]
log-bin=mysql-bin.log
server_id=100
binlog_row_image=full
binlog-format=row
expire_logs_days =10

Octave Psychtoolbox crashing when the screen is called to be displayed

I am still new in Octave with some experience in Python and I was requested to do a test in a stimulus application in order to find if there are errors in the code (not developed by me) and try to fix them and run the application. The issue in the code is at the time to open the screen as follows the implementation (these ones where I commented (%) are what already did before, and these ones below the comments are my correction) the code is large with 825 lines:
%% Get subject name from the user
prompt = {'Enter Subject Number:','Enter Subject Name:','Enter Todays Date:','Left or Right:'};
dlgtitle = 'Information Input';
%dims = [1 50];
dims = [1 50;1 50;1 50;1 50];
opts = 'on';
answer = inputdlg(prompt,dlgtitle,dims);
%waitfor(answer);
waitfor(dims);
%% TO Excel
% Get the name of the file that the user wants to save.
%startingFolder = userpath;
startingFolder = 'C:\Users\Vtr\Downloads\Final-task-981203\Final-task-981203\Task-981203';
defaultFileName = fullfile(startingFolder, '*.xlsx');
[baseFileName, folder] = uiputfile(defaultFileName, 'Specify a filename ');
if baseFileName == 0
% User clicked the Cancel button.
return;
end
% Combine the folder and base file name into the full file name.
fullFileName = fullfile(folder, baseFileName);
%% LOAD text initial setting
%% Create 1st screen : (Loading and Wait)text screen
% Open an on screen window - purple screen
[window, windowRect] = PsychImaging('OpenWindow', screenNumber, [0.5 0.5 1]);
% Set up alpha-blending for smooth (anti-aliased) lines
Screen('BlendFunction', window, 'GL_SRC_ALPHA', 'GL_ONE_MINUS_SRC_ALPHA');
After that, in the first part is created a form to insert subject nÂș, subject name, today's date and left or right, and after that to specify a filename. Made these two parts it appears a purple screen with the mouse's cursor rotating non-stop and if I try to click with the mouse an error alert box "octave-gui.exe has stopped working" appears on. I already tried to run this code on my Virtualbox-VM (Ubuntu 16.04 LTS), firstly, it notifies that my screen is being tested and calibrated and after, it appears another error screen on the terminal:
PTB-INFO: OS support status: Linux 4.15.0-55-generic Supported.
PTB-INFO: Type 'PsychtoolboxVersion' for more detailed version information.
PTB-INFO: Most parts of the Psychtoolbox distribution are licensed to you under terms of the MIT License, with
PTB-INFO: some restrictions. See file 'License.txt' in the Psychtoolbox root folder for the exact licensing conditions.
PTB-WARNING: FAILED to enable synchronization to vertical retrace!
PTB-WARNING: FAILED to enable synchronization to vertical retrace (System ignored setting [Req 1 != Actual 0])!
PTB-INFO: Failed to enable realtime-scheduling [Operation not permitted]!
PTB-INFO: You need to run Matlab or Octave with root-privileges, or run the script PsychLinuxConfiguration once for this to work.
PTB-INFO: See /usr/share/doc/psychtoolbox-3-common/README.Debian to make this work.
PTB-WARNING: Couldn't even collect one single valid flip interval sample! Sanity range checks failed!
PTB-WARNING: Could be a system bug or a temporary timing problem. Retrying the procedure might help if
PTB-WARNING: the latter is the culprit.
WARNING: VBL Calibration run No. 1 failed. Retrying...
PTB-INFO: Failed to enable realtime-scheduling [Operation not permitted]!
PTB-INFO: You need to run Matlab or Octave with root-privileges, or run the script PsychLinuxConfiguration once for this to work.
PTB-INFO: See /usr/share/doc/psychtoolbox-3-common/README.Debian to make this work.
PTB-WARNING: Couldn't even collect one single valid flip interval sample! Sanity range checks failed!
PTB-WARNING: Could be a system bug or a temporary timing problem. Retrying the procedure might help if
PTB-WARNING: the latter is the culprit.
WARNING: VBL Calibration run No. 2 failed. Retrying...
WARNING: Will enable VBL busy wait-workaround before trying final VBL Calibration run No. 3.
WARNING: This will hopefully work-around graphics driver bugs of the GPU sync-to-retrace mechanism. Cross your fingers!
PTB-INFO: Failed to enable realtime-scheduling [Operation not permitted]!
PTB-INFO: You need to run Matlab or Octave with root-privileges, or run the script PsychLinuxConfiguration once for this to work.
PTB-INFO: See /usr/share/doc/psychtoolbox-3-common/README.Debian to make this work.
PTB-WARNING: Couldn't even collect one single valid flip interval sample! Sanity range checks failed!
PTB-WARNING: Could be a system bug or a temporary timing problem. Retrying the procedure might help if
PTB-WARNING: the latter is the culprit.
PTB-INFO: OpenGL-Renderer is VMware, Inc. :: SVGA3D; build: RELEASE; LLVM; :: 2.1 Mesa 18.0.5
PTB-INFO: VBL startline = 664 , VBL Endline = -1
PTB-INFO: Beamposition queries unsupported or defective on this system. Using basic timestamping as fallback.
PTB-INFO: Timestamps returned by Screen('Flip') will be therefore less robust and accurate.
PTB-INFO: Measured monitor refresh interval from VBLsync = 0.000000 ms [inf Hz]. (0 valid samples taken, stddev=10000000.000000 ms.)
PTB-INFO: Reported monitor refresh interval from operating system = 16.667222 ms [59.998001 Hz].
PTB-INFO: Small deviations between reported values are normal and no reason to worry.
WARNING: Couldn't compute a reliable estimate of monitor refresh interval! The trouble with VBL syncing?!?
WARNING: Mismatch between measured monitor refresh interval and interval reported by the operating system.
This indicates massive problems with VBL sync.
----- ! PTB - ERROR: SYNCHRONIZATION FAILURE ! -----
One or more internal checks (see Warnings above) indicate that synchronization
of Psychtoolbox to the vertical retrace (VBL) is not working on your setup.
This will seriously impair proper stimulus presentation and stimulus presentation timing!
Please read 'help SyncTrouble' for information about how to solve or work-around the problem.
You can force Psychtoolbox to continue, despite the severe problems, by adding the command
Screen('Preference', 'SkipSyncTests', 1); at the top of your script, if you really know what you are doing.
error: Screen: See error message printed above.
error: Called from
PsychImaging at line 2240 column 26
Naderi_test_task_v_main_1203_v_2 at line 110 column 20
I would like to know if the issue is in my computer configurations that don't support Psychtoolbox resources or any configuration that I need to change.

neo4j shell -creating edges from .csv

I am using comma separated value file to create nodes and edges in a Neo4j database. The commands which create nodes run with no issue. The attempt to create edges fails with this error:
Exception in thread "GC-Monitor" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "GC-Monitor"
Further, in the output from the commands there was this:
neo4j-sh (?)$ using periodic commit 400 load csv with headers from 'file://localhost/tmp/vm2set3.csv' as line match (u:VM {id: line.vm_id}),(s:VNIC {id: line.set3_id}) create (u)-[:VNIC]->(s);
SystemException: Kernel has encountered some problem, please perform neccesary action (tx recovery/restart)
neo4j-sh (?)$
SystemException: Kernel has encountered some problem, please perform neccesary action (tx recovery/restart)
neo4j-sh (?)$ using periodic commit 400 load csv with headers from 'file://localhost/tmp/unix2switch.csv' as line match (u:UNIX {id: line.intf_id}),(s:switch {id: line.set2a_id}) create (u)-[:cable]->(s);
SystemException: Kernel has encountered some problem, please perform neccesary action (tx recovery/restart)
neo4j-sh (?)$
My shell script is:
cat /home/ES2Neo/2.1/neo4j_commands.cql | /export/neo4j-community-2.1.4/bin/neo4j-shell -path /export/neo4j-community-2.1.4/data/graph.db > /tmp/na.out
The commands are like this:
load csv WITH HEADERS from 'file://localhost/tmp/intf.csv' AS line CREATE (:UNIX {id: line.id, MAC: line.MAC ,BIA: line.BIA ,host: line.host,name: line.name});
for nodes, and
using periodic commit 400 load csv with headers from 'file://localhost/tmp/unix2switch.csv' as line match (u:UNIX {id: line.intf_id}),(s:switch {id: line.set2a_id}) create (u)-[:cable]->(s);
for edges.
The csv input files look like this:
"intf_id","set2a_id"
"100321","6724919"
"125850","6717849"
"158249","6081895"
"51329","5565380"
"57248","6680663"
"235196","6094139"
"229242","4800249"
"225630","6661742"
"183281","4760022"
Is there something I am doing wrong? Is there something in Neo4j configuration I need to check? Thanks.
The problem is that you're running out of memory for loading the data into the database.
Take a look at this blog post which goes into a number of details about how to load CSV data in successfully.
In particular, here's the key bit from the blog post you should pay attention to.
The more memory you have the faster it will import your data.
So make sure to edit conf/neo4j-wrapper.conf and set:
# Uncomment these lines to set specific initial and maximum
# heap size in MB.
wrapper.java.initmemory=4096
wrapper.java.maxmemory=4096
In conf/neo4j.properties set:
# Default values for the low-level graph engine
neostore.nodestore.db.mapped_memory=50M
neostore.relationshipstore.db.mapped_memory=500M
neostore.propertystore.db.mapped_memory=100M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=0M

warning message at the prompt

I'm trying to simulate a testbench. I'm not getting the waveforms also i'm getting the following warning message at the prompt. Is it because of the=is warning that my code does not simulate?
** Warning: (vsim-WLF-5000) WLF file currently in use: vsim.wlf
# File in use by: Hostname: ProcessID: 0
# Attempting to use alternate WLF file "./wlftazxa4k".
# ** Warning: (vsim-WLF-5001) Could not open WLF file: vsim.wlf
# Using alternate file: ./wlftazxa4k
run
I'm also includng my testbench as follows:
module dec_tb;
reg [63:0] FROM_IF_ID;
reg CLK;
wire [117:0] TO_ID_HMIC;
integer k=0;
inst_decode id(.from_if_id(FROM_IF_ID),.clk(CLK),.to_id_hmic(TO_ID_HMIC));
initial
begin
$monitor($time,"clk=%b, fifid=%b, tidhm=%b",CLK,FROM_IF_ID,TO_ID_HMIC);
$display("qf");
CLK= 0;
FROM_IF_ID[35:32]=4'b1100;
FROM_IF_ID[63:36]=28'b0000_10000_00100_01000_00010_0001;
end
always
begin
#10 CLK= ~CLK;
end
always #(posedge CLK)
begin
$display (" TO_ID_HMIC= %b", TO_ID_HMIC);
FROM_IF_ID[k] =~FROM_IF_ID[k];
k=k+1;
#500 $finish;
end
endmodule
If that's the only message, then the simulation ran, and you can find the waves in the specified alternate file (wlftazxa4k).
If you want to fix the problem so the waves show up in vsim.wlf, here are a few things to try:
Make sure you don't have any stray modelsim processes running
Make sure you don't have vsim.wlf open in a waveform viewer
Delete vsim.wlf manually and rerun
I had this problem also. As it turns out, the directory that my vsim files were located in was full (school partition, 600mb allowed space). After clearing out some old files i had lying around, the program worked fine.
Your your quiz:
Run you simulation
Quit your simulation with gui or command: quit -sim
Goto step 1 and have fun.
You can remove cache file create by modelsim.

restart multi-threaded perl script and close mysql connection

I have a Perl script that reads a command file and restarts itself if necessary by doing:
myscript.pl:
exec '/home/foo/bin/myscript.pl';
exit(0);
Now, this works fine except for one issue. The thread that reads the command file does not have access to the DBI handle I use. And over a number of restarts I seem to build up the number of open mysql connections till I get the dreaded "Too Many Connections" error. The DBI spec says:
"Because of this (possibly temporary) restriction, newly created threads must make their own connections to the database. Handles can't be shared across threads."
Any way to close connections or perhaps a different way to restart the script?
Use a flag variable that is shared between threads. Have the command line reading thread set the flag to exit, and the thread holding the DB handle release it and actually do the re-exec:
#!/usr/bin/perl
use threads;
use threads::shared;
use strict; use warnings;
my $EXIT_FLAG :shared;
my $db_thread = threads->create('do_the_db_thing');
$db_thread->detach;
while ( 1 ) {
sleep rand 10;
$EXIT_FLAG = 1 if 0.05 > rand or time - $^T > 20;
}
sub do_the_db_thing {
until ( $EXIT_FLAG ) {
warn sprintf "%d: Working with the db\n", time - $^T;
sleep rand 5;
}
# $dbh->disconnect; # here
warn "Exit flag is set ... restarting\n";
exec 'j.pl';
}
You could try registering an atexit function to close the DBI handle at the point where it is opened, and then use fork & exec to restart the script rather then just exec. The parent would then call exit, invoking the atexit callback to close the DBI handle. The child could re-exec itself normally.
Edit: After thinking for a couple more minutes, I believe you could skip the atexit entirely because the handle would be closed automatically upon the parent's exit. Unless, of course, you need to do a more complex operation upon closing the DB handle than a simple filehandle close.
my $pid = fork();
if (not defined $pid) {
#Could not fork, so handle the error somehow
} elsif ($pid == 0) {
#Child re-execs itself
exec '/home/foo/bin/myscript.pl';
} else {
#Parent exits
exit(0);
}
If you expect a lot of connections, you probably want DBI::Gofer to act as a DBI proxy for you. You create as many connections in as many scripts as you like, and DBI::Gofer shares them when it can.