Does the MySql binlog contain transaction boundaries? - mysql

If I'm reading a MySql binlog, can I get an indication of which statements occur in the same transaction?

There's nothing built-in yet, but perhaps this page will provide some help. They provide an awk script that will parse the binary log and provide transaction details, for row-based replication at least. We don't like link-only answers, so I'll post the script itself here:
startdate="2015-01-12 21:40:00"
stopdate="2015-01-12 21:45:00"
logfile="mysqld-bin.000023"
mysqlbinlog --base64-output=decode-rows -vv --start-datetime="$startdate" --stop-datetime="$stopdate" $logfile | awk \
'BEGIN {s_type=""; s_count=0;count=0;insert_count=0;update_count=0;delete_count=0;flag=0;} \
{if(match($0, /#15.*Table_map:.*mapped to number/)) {printf "Timestamp : " $1 " " $2 " Table : " $(NF-4); flag=1} \
else if (match($0, /(### INSERT INTO .*..*)/)) {count=count+1;insert_count=insert_count+1;s_type="INSERT"; s_count=s_count+1;} \
else if (match($0, /(### UPDATE .*..*)/)) {count=count+1;update_count=update_count+1;s_type="UPDATE"; s_count=s_count+1;} \
else if (match($0, /(### DELETE FROM .*..*)/)) {count=count+1;delete_count=delete_count+1;s_type="DELETE"; s_count=s_count+1;} \
else if (match($0, /^(# at) /) && flag==1 && s_count>0) {print " Query Type : "s_type " " s_count " row(s) affected" ;s_type=""; s_count=0; } \
else if (match($0, /^(COMMIT)/)) {print "[Transaction total : " count " Insert(s) : " insert_count " Update(s) : " update_count " Delete(s) : " \
delete_count "] \n+----------------------+----------------------+----------------------+----------------------+"; \
count=0;insert_count=0;update_count=0; delete_count=0;s_type=""; s_count=0; flag=0} } '

Related

Vim & NeoVim Cannot Detect Django HTML FileType

I'm using Vim 8.1 and NeoVim 0.4.3. When I'm editing Django template html files neither could correctly detect the file type and hence no color or syntax highlighting at all. I'm pretty sure filetype on because this is the output:
filetype dection:ON plugin:ON indent:ON
I followed this issue and forcing the filetype to htmldjango (at the bottom of the .vimrc file I post down below) did work, but I don't think it's ideal. Could anybody please help me with this?
My .vimrc:
" macOS version
set rtp+=/usr/local/opt/fzf
call plug#begin('~/.vim/plugged')
Plug 'junegunn/vim-easy-align'
" tmux
Plug 'christoomey/vim-tmux-navigator'
Plug 'tmux-plugins/vim-tmux'
Plug 'benmills/vimux'
" Coc
Plug 'neoclide/coc.nvim', {'branch': 'release'}
Plug 'tpope/vim-fugitive'
" fzf
Plug '/usr/local/opt/fzf'
Plug 'junegunn/fzf.vim'
" Ack search tool
Plug 'mileszs/ack.vim'
" Indentation
Plug 'Yggdroot/indentLine'
" Python autoformat
Plug 'psf/black'
" Django HTML
Plug 'tweekmonster/django-plus.vim'
" comment
Plug 'scrooloose/nerdcommenter'
Plug 'tpope/vim-sensible'
Plug 'tpope/vim-sleuth'
" gitgutter
Plug 'airblade/vim-gitgutter'
" emmet
Plug 'mattn/emmet-vim'
" Ale
Plug 'w0rp/ale'
" lightline
Plug 'itchyny/lightline.vim'
" Eslint
Plug 'vim-syntastic/syntastic'
" EcmaScript and JSX
Plug 'pangloss/vim-javascript'
Plug 'maxmellon/vim-jsx-pretty'
" TypeScript
Plug 'leafgarland/typescript-vim'
Plug 'Quramy/tsuquyomi'
Plug 'Shougo/vimproc.vim'
" Elm
Plug 'elmcast/elm-vim'
" Go format
Plug 'fatih/vim-go'
" NERDTree
Plug 'scrooloose/nerdtree' ", {'on': 'NERDTreeToggle'}
Plug 'Xuyuanp/nerdtree-git-plugin'
" Colorscheme
Plug 'morhetz/gruvbox'
Plug 'octol/vim-cpp-enhanced-highlight'
Plug 'nanotech/jellybeans.vim'
" markdown
Plug 'godlygeek/tabular'
Plug 'plasticboy/vim-markdown'
Plug 'prettier/vim-prettier', {
\ 'do': 'yarn install',
\ 'branch': 'release/1.x',
\ 'for': [
\ 'javascript',
\ 'typescript',
\ 'css',
\ 'less',
\ 'scss',
\ 'json',
\ 'graphql',
\ 'markdown',
\ 'vue',
\ 'lua',
\ 'swift' ] }
" plugin from http://vim-scripts.org/vim/scripts.html
" Git plugin not hosted on GitHub
Plug 'git://git.wincent.com/command-t.git'
" The sparkup vim script is in a subdirectory of this repo called vim.
" Pass the path to set the runtimepath properly.
Plug 'rstacruz/sparkup', {'rtp': 'vim/'}
" All of your Plugins must be added before the following line
call plug#end() " required
" To ignore plugin indent changes, instead use:
" Put your non-Plugin stuff after this line
set mouse=a
set hidden
set cmdheight=2
set nobackup
set nowritebackup
" don't give |ins-completion-menu| messages.
set shortmess+=c
" always show signcolumns
set signcolumn=yes
" Use tab for trigger completion with characters ahead and navigate.
" Use command ':verbose imap <tab>' to make sure tab is not mapped by other plugin.
" use <tab> for trigger completion and navigate to the next complete item
function! s:check_back_space() abort
let col = col('.') - 1
return !col || getline('.')[col - 1] =~ '\s'
endfunction
inoremap <silent><expr> <TAB>
\ pumvisible() ? "\<C-n>" :
\ <SID>check_back_space() ? "\<TAB>" :
\ coc#refresh()
inoremap <expr> <TAB> pumvisible() ? "\<C-n>" : "\<TAB>"
inoremap <expr> <S-TAB> pumvisible() ? "\<C-p>" : "\<S-TAB>"
augroup mygroup
autocmd!
" Setup formatexpr specified filetype(s).
autocmd FileType typescript,json setl formatexpr=CocAction('formatSelected')
" Update signature help on jump placeholder
autocmd User CocJumpPlaceholder call CocActionAsync('showSignatureHelp')
augroup end
" Use K to show documentation in preview window
nnoremap <silent> K :call <SID>show_documentation()<CR>
function! s:show_documentation()
if (index(['vim','help'], &filetype) >= 0)
execute 'h '.expand('<cword>')
else
call CocAction('doHover')
endif
endfunction
set updatetime=100
set timeout timeoutlen=3000 ttimeoutlen=100
set number
set tabstop=4
set smarttab
set softtabstop=4
set expandtab
set textwidth=80
set colorcolumn=81
set wrapmargin=0
set formatoptions=croqlt12
set wrap
set shiftwidth=2
set autoindent
set smartindent
set showcmd
set cursorline
set wildmenu
set lazyredraw
set showmatch
set incsearch
set hlsearch
set foldenable
set cindent
set shell=/bin/bash
set viminfo='100,<1000,s100,h
colorscheme gruvbox
set background=dark
let g:gruvbox_contrast_dark='hard'
set foldlevelstart=99
set foldnestmax=10
nnoremap <space> za
" new line without insert mode
nmap <S-Enter> O<Esc>j
nmap <CR> o<Esc>k
set foldmethod=indent
inoremap ( ()<Esc>i
inoremap (<CR> (<CR>)<C-o>O
inoremap [ []<Esc>i
inoremap [<CR> [<CR>]<C-o>O
"inoremap < <><Esc>i
inoremap { {}<Esc>i
inoremap ` ``<Esc>i
inoremap {<CR> {<CR>}<C-o>O
inoremap `<CR> `<CR>`<C-o>O
"inoremap <C-Return> <CR><CR><C-o>k<Tab>
inoremap " ""<Esc>i
inoremap ' ''<Esc>i
" navigation
nnoremap <C-J> <C-W><C-J>
nnoremap <C-K> <C-W><C-K>
nnoremap <C-L> <C-W><C-L>
nnoremap <C-H> <C-W><C-H>
set splitbelow
set splitright
" fugitive
autocmd QuickFixCmdPost *grep* cwindow
set diffopt+=vertical
" Vimux
" Prompt for a command to run
map <Leader>vp :VimuxPromptCommand<CR>
" Run last command executed by VimuxRunCommand
map <Leader>vl :VimuxRunLastCommand<CR>
" Coc config
set completeopt+=preview
set completeopt+=menuone
set completeopt+=noselect
autocmd CursorHold * silent call CocActionAsync('highlight')
autocmd FileType json syntax match Comment +\/\/.\+$+
let g:ale_linters = {
\ 'sh': ['language_server'],
\ }
autocmd User CocJumpPlaceholder call CocActionAsync('showSignatureHelp')
" gitgutter
let g:gitgutter_max_signs = 999
" NERD Commenter
" Add spaces after comment delimiters by default
let g:NERDSpaceDelims = 1
" Use compact syntax for prettified multi-line comments
let g:NERDCompactSexyComs = 1
" Align line-wise comment delimiters flush left instead of following code indentation
let g:NERDDefaultAlign = 'left'
" Set a language to use its alternate delimiters by default
let g:NERDAltDelims_java = 1
" Add your own custom formats or override the defaults
let g:NERDCustomDelimiters = { 'c': { 'left': '/**','right': '*/' } }
" Allow commenting and inverting empty lines (useful when commenting a region)
let g:NERDCommentEmptyLines = 1
" Enable trimming of trailing whitespace when uncommenting
let g:NERDTrimTrailingWhitespace = 1
" Enable NERDCommenterToggle to check all selected lines is commented or not
let g:NERDToggleCheckAllLines = 1
function! Formatonsave()
let l:formatdiff = 1
pyf ~/clang-format.py
endfunction
" lightline colorscheme
function! CocCurrentFunction()
return get(b:, 'coc_current_function', '')
endfunction
let g:lightline = {
\ 'colorscheme': 'one',
\ 'active': {
\ 'left': [ [ 'mode', 'paste' ],
\ [ 'cocstatus', 'currentfunction', 'readonly', 'filename', 'modified' ] ]
\ },
\ 'component_function': {
\ 'cocstatus': 'coc#status',
\ 'currentfunction': 'CocCurrentFunction'
\ },
\ }
" fzf search
" Default fzf layout
" - down / up / left / right
let g:fzf_layout = { 'down': '~30%' }
" In Neovim, you can set up fzf window using a Vim command
" let g:fzf_layout = { 'window': 'enew' }
" let g:fzf_layout = { 'window': '-tabnew' }
" let g:fzf_layout = { 'window': '10split' }
let g:fzf_action = {
\ 'ctrl-t': 'tab split',
\ 'ctrl-i': 'split',
\ 'ctrl-s': 'vsplit' }
autocmd! FileType fzf set laststatus=0 noshowmode noruler
\| autocmd BufLeave <buffer> set laststatus=2 showmode ruler
command! -bang -nargs=* Rg
\ call fzf#vim#grep(
\ 'rg --hidden --column --line-number --no-heading --color=always --smart-case '.shellescape(<q-args>), 1,
\ <bang>0 ? fzf#vim#with_preview('up:60%')
\ : fzf#vim#with_preview('right:50%:hidden', '?'),
\ <bang>0)
nnoremap <C-g> :Rg<Cr>
" Global line completion (not just open buffers. ripgrep required.)
inoremap <expr> <c-x><c-l> fzf#vim#complete(fzf#wrap({
\ 'prefix': '^.*$',
\ 'source': 'rg -n ^ --color always',
\ 'options': '--ansi --delimiter : --nth 3..',
\ 'reducer': { lines -> join(split(lines[0], ':\zs')[2:], '') }}))
" [Buffers] Jump to the existing window if possible
" let g:fzf_buffers_jump = 1
nnoremap <silent> <Leader>s :call fzf#run({
\ 'down': '40%',
\ 'sink': 'botright split' })<CR>
" Open files in vertical horizontal split
nnoremap <silent> <Leader>v :call fzf#run({
\ 'right': winwidth('.') / 2,
\ 'sink': 'vertical botright split' })<CR>
let g:ackprg = 'rg --vimgrep --no-heading'
cnoreabbrev ag Ack
cnoreabbrev aG Ack
cnoreabbrev Ag Ack
cnoreabbrev AG Ack
let g:vim_jsx_pretty_colorful_config = 1
let g:typescript_indent_disable = 0
let g:typescript_ignore_browserwords = 0
autocmd BufWritePre *.h,*.cc,*.c,*.cpp call Formatonsave()
map <C-K> :pyf ~/clang-format.py<cr>
imap <C-K> <c-o>:pyf ~/clang-format.py<cr>
let g:autopep8_aggressive=1
let g:autopep8_disable_show_diff=1
let g:autopep8_on_save=1
let g:autopep8_max_line_length=120
set statusline+=%#warningmsg#
set statusline+=%{SyntasticStatuslineFlag()}
set statusline+=%{coc#status()}
set statusline+=%*
let g:syntastic_python_checkers = ['pylint']
let g:syntastic_always_populate_loc_list = 1
let g:syntastic_auto_loc_list = 1
let g:syntastic_check_on_open = 1
let g:syntastic_check_on_wq = 1
let g:syntastic_javascript_checkers = ['eslint']
let g:syntastic_javascript_eslint_exe = 'npm run lint --'
let g:elm_syntastic_show_warnings = 1
" NERDTree
let g:NERDTreeWinSize = 27
" autocmd vimenter * NERDTree
autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTree") && b:NERDTree.isTabTree()) | q | endif
map <C-n> :NERDTreeToggle<CR>
" start vim/nvim with $vim or $nvim, NOT $vim . or $nvim .
autocmd StdinReadPre * let s:std_in=1
autocmd VimEnter * if argc() == 0 && !exists("s:std_in") | NERDTree | endif
autocmd StdinReadPre * let s:std_in=1
autocmd VimEnter * if argc() == 1 && isdirectory(argv()[0]) && !exists("s:std_in") | exe 'NERDTree' argv()[0] | wincmd p | ene | endif
" Please customize this:
let g:NERDTreeIndicatorMapCustom = {
\ "Modified" : "✹",
\ "Staged" : "✚",
\ "Untracked" : "✭",
\ "Renamed" : "➜",
\ "Unmerged" : "═",
\ "Deleted" : "✖",
\ "Dirty" : "✗",
\ "Clean" : "✔︎",
\ 'Ignored' : '☒',
\ "Unknown" : "?"
\ }
let g:prettier#config#print_width = 80
let g:prettier#config#html_whitespace_sensitivity = 'css'
let g:prettier#config#trailing_comma = 'all'
let g:prettier#config#semi = 'true'
let g:prettier#exec_cmd_async=1
let g:prettier#autoformat = 0
autocmd BufWritePre *.js,*.jsx,*.mjs,*.ts,*.tsx,*.css,*.less,*.scss,*.json,*.graphql,*.md,*.vue,*.yaml PrettierAsync
" Auto-format Go
autocmd BufWritePre *.go :call CocAction('runCommand', 'editor.action.organizeImport')
" Python Black
let g:black_linelength = 120
let g:black_skip_string_normalization = 1
" Auto run Black
autocmd BufWritePre *.py execute ':Black'
" djangohtml
au BufNewFile,BufRead *.html set filetype=htmldjango
Turned out it's the vim-prettier plugin causing the conflict. Disabling that plugin solved the issue.

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html.
My awk script looks like :
echo "<p><h2>FRAME : $test</h2></p>"
echo "<table>"
for fn in /var/www/cgi-bin/LPAR_MAP/*;
do
echo "<td>"
echo "<PRE>"
awk -F',|;' -v test="$test" '
NR==1 {
split(FILENAME ,a,"[-.]");
}
$0 ~ test {
if(!header++){
print "DATE ========================== : " a[4]
}
print ""
print "LPARS :" $2
print "RAM : " $5
print "CPU 1 : " $6
print "CPU 2 : " $7
print ""
print ""
}' $fn;
echo "</PRE>"
echo "</td>"
done
echo "</table>"
This script allow to analyze 276 csv files that looks like :
MO2PPC20;mo2vio20b;Running;VIOS 2.2.5.20;7;1.0;2;DefaultPool;shared;uncap;192
MO2PPC20;mo2vio20a;Running;VIOS 2.2.5.20;7;1.0;2;DefaultPool;shared;uncap;192
MO2PPC21;mplaix0311;Running;AIX 7.1 7100-05-02-1832;35;0.6;4;DefaultPool;shared;uncap;64
MO2PPC21;miaibv194;Running;AIX 6.1 6100-09-11-1810;11;0.2;1;DefaultPool;shared;uncap;64
MO2PPC21;mplaix0032;Running;AIX 6.1 6100-09-11-1810;105;4.0;11;DefaultPool;shared;uncap;128
MO2PPC21;mplaix0190;Running;Unknown;243;4.9;30;DefaultPool;shared;uncap;128
MO2PPC21;mo2vio21b;Running;VIOS 2.2.6.10;6;1.5;3;DefaultPool;shared;uncap;192
MO2PPC21;miaibv238;Running;AIX 7.1 7100-05-02-1810;10;0.5;1;DefaultPool;shared;uncap;64
MO2PPC21;mo2vio21a;Running;VIOS 2.2.6.10;6;1.5;3;DefaultPool;shared;uncap;192
MO2PPC21;miaibv193;Running;AIX 6.1 6100-09-11-1810;12;0.2;1;DefaultPool;shared;uncap;64
MO1PPC17;miaibe03;Running;AIX 5.2 5200-10-08-0930;25;null;3;null;ded;share_idle_procs;null
MO1PPC17;miaiba12;Running;AIX 5.2 5200-10-08-0930;17;null;2;null;ded;share_idle_procs;null
MO1PPC17;miaibf03;Running;AIX 5.2 5200-10-08-0930;30;null;3;null;ded;share_idle_procs;null
MO1PPC17;miaibc05;Running;AIX 5.2 5200-10-08-0930;40;null;2;null;ded;share_idle_procs;null
And to display them in my CGI like this :
The numbers of columns is equal at the number of csv to analyze.
As you can see in the screenshot, some lines are sometimes the same in each csv files
The idea is to delete the lines that are the same in all my csv files.
I know the awk command :
awk '!a[$0]++'
But this command needs a file to be achieved.
Do you think it's possible to put this command in my awk script ?
Read all files at once with awk and add condition :
awk -F',|;' -v test="$test" '
BEGIN{
print "<table>"
}
FNR==1 {
if(close_tag++){
print "</PRE>"
print "<td>"
}
print "<td>"
print "<PRE>"
split(FILENAME ,a,"[-.]");
}
($0 ~ test) && (!dup[$0]++) {
if(!header++){
print "DATE ========================== : " a[4]
}
print ""
print "LPARS :" $2
print "RAM : " $5
print "CPU 1 : " $6
print "CPU 2 : " $7
print ""
print ""
}
END{
print "</PRE>"
print "</td>"
print "</table>"
}' /var/www/cgi-bin/LPAR_MAP/*

How to send a html file using mails in shell script?

I want to store my matrix's data in a html file and send a email in outlook ,my code looks as follows:
printf "<!DOCTYPE html>"
printf "<html>"
printf "<head>"
printf "<style>"
printf "</style>"
printf "</head>"
printf "<body>"
printf "<table>"
printf "<tr>"
printf "<th>Total</th>"
printf "<th>StillFail</th>"
printf "<th>Pass</th>"
printf "<th>ScriptError</th>"
printf "<th>APIName</th>"
printf "</tr>"
printf "<tr>"
echo
for ((j=1;j<=num_rows;j++)) do
printf "<tr>"
for ((i=1;i<=num_columns;i++)) do
printf "<td>"
printf "${matrix[$j,$i]}"
printf "</td>"
printf "</tr>"
done
echo
done
printf "</tr>"
printf "</table>"
printf "</body>"
printf "</html>"
#mailx -a 'Content-Type: html' -s "my subject" test#example.com < output.html
mailx -s "TESTING MAIL"</home/test/example/output.html "test#example.com"
I want my output as a well aligned table. Can someone help me on this? TIA
You need to append content-type in header it can be done with -a flag in mailx
comment your last line and try this
mailx -a 'Content-Type: text/html' -s "Test MAIL" test#example.com</home/test/example/output.html
Edit :
As per the OP End error : content-type:texthtml no such file or
directory
To install mailutils in mac
Press Command+Space and type Terminal and press enter/return key.
Run in Terminal app:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null
and press enter/return key. Wait for the command to finish.
Run:
brew install mailutils
reference : http://macappstore.org/mailutils/
Maybe you can use this for a base:
(
echo "To: ADRESSES"
echo "Subject: SUBJECT"
echo "Content-Type: text/html"
echo
echo "HTML code"
) | /usr/sbin/sendmail -F "NameOfSender" -t

Git statistics API JSON

I want to get the number of commits per day per hour. With the following command, I am able to get the output in json format. However, I would like to know if I can add the key to the values in json format using command line?
curl https://api.github.com/repos/test/myrepo/stats/punch_card
Current output:
[
0,
2,
32
]
Expected output:
[
day: 0,
hour: 2,
commits: 32
]
Since you haven't specified anything beyond "command line", I'm assuming you want a bash-based solution. This simple (though kind of ugly) script will do what you want, while maintaining indentation (apart from the closing square bracket of the overall response):
#!/bin/bash
resp=$(curl https://api.github.com/repos/test/myrepo/stats/punch_card)
nextPref=""
for val in $resp
do
echo "$nextPref $val"
if [[ $val == "[" && $nextPref == "" ]]
then
nextPref=" "
elif [[ $val == "[" && $nextPref == " " ]]
then
nextPref=" day:"
elif [[ $nextPref == " day:" ]]
then
nextPref=" hour:"
elif [[ $nextPref == " hour:" ]]
then
nextPref=" commits:"
elif [[ $nextPref == " commits:" ]]
then
nextPref=" "
fi
done

inserting data into MYSQL tables by using shell script

I am a beginner in shell script. I am trying to store the output of a LINUX command in MySQL tables. I need to select the partition details in a column and used % in another column. I nearly made it but i get the output in a single column.In table test, disk is a column and used is another column. My desired output is
**DISK** **USED**
filesystem 45%
but my actual output is like
**DISK USED**
filesystem
45%
My code:
df -h | tee /home/abcd/test/monitor.text;
details=$(awk '{ print $1 } ' monitor.text);
echo $details;
used=$(awk '{ print $5}' monitor.text);
echo $used;
mysql test<<EOF;
INSERT INTO test_1 (details,used) VALUES ('$details','$used');
EOF
Please give me the correct code for the desired output. Thank you in advance.
Here is a script which captures the value of those 2 columns fine You can do the inserts?
#!/bin/sh
df -h | awk '{ print$1 " " $5} ' > monitor.txt
exec<monitor.txt
value=0
while read line
do
col1=`echo $line | cut -f1 -d " " `
col2=`echo $line | cut -f2 -d " " `
echo $col1
echo $col2 ;
done
Plug your insert statements inside the do-done loop.