Sphinx Error: failed to merge index - mysql

While merging index with delta on Sphinx, I got this error:
~: /usr/local/bin/indexer --merge myindex myindexDelta --rotate;
Sphinx 2.0.6-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc ( http://sphinxsearch.com )
using config file '/usr/local/etc/sphinx.conf'...
merging index 'myindexDelta' into index 'myindex'...
read 414.6 of 414.6 MB, 100.0% done
FATAL: failed to merge index 'myindexDelta' into index 'myindex': failed to open /server/sphinx/data/myindex.sps: No such file or directory
My configuration on sphinx.conf is as following
source myindex
{
type = mysql
sql_host = localhost
sql_user = db
sql_pass =
sql_db = db
sql_query_pre = SET SESSION query_cache_type=OFF
sql_query_pre = REPLACE INTO sph_counter SELECT 1, MAX(id) FROM mytable
sql_query_pre = SET NAMES utf8
sql_query = \
SELECT id,title FROM mytable \
WHERE id<=( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
sql_ranged_throttle = 0
}
source myindexDelta : myindex
{
sql_query_pre = SET SESSION query_cache_type=OFF
sql_query_pre = SET NAMES utf8
sql_query = \
SELECT id,title FROM mytable \
WHERE id > ( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
}
index myindex
{
source = myindex
path = /server/sphinx/data/myindex
min_word_len = 3
min_infix_len = 0
}
index myindexDelta : myindex
{
source = myindexDelta
path = /server/sphinx/data/myindexDelta
min_word_len = 3
min_infix_len = 0
}
indexes files listings with permissions:
~: ls -lh /server/sphinx/data/
-rw-r--r-- 1 root root 0 Nov 11 21:40 myindexDelta.spa
-rw-r--r-- 1 root root 290K Nov 11 21:40 myindexDelta.spd
-rw-r--r-- 1 root root 328 Nov 11 21:40 myindexDelta.sph
-rw-r--r-- 1 root root 106K Nov 11 21:40 myindexDelta.spi
-rw-r--r-- 1 root root 0 Nov 11 21:40 myindexDelta.spk
-rw------- 1 root root 0 Nov 11 21:40 myindexDelta.spl
-rw-r--r-- 1 root root 0 Nov 11 21:40 myindexDelta.spm
-rw-r--r-- 1 root root 223K Nov 11 21:40 myindexDelta.spp
-rw-r--r-- 1 root root 1 Nov 11 21:40 myindexDelta.sps
-rw-r--r-- 1 root root 0 Jul 3 21:17 myindex.spa
-rw-r--r-- 1 root root 7.0G Jul 3 23:54 myindex.spd
-rw-r--r-- 1 root root 290 Jul 3 23:54 myindex.sph
-rw-r--r-- 1 root root 397M Jul 3 23:54 myindex.spi
-rw-r--r-- 1 root root 0 Jul 3 23:54 myindex.spk
-rw------- 1 root root 0 Nov 11 21:08 myindex.spl
-rw-r--r-- 1 root root 0 Jul 3 21:17 myindex.spm
-rw-r--r-- 1 root root 9.2G Jul 3 23:54 myindex.spp
I am sure the code explains everything, adding description is not necessary.

I'm guessing that the original 'myindex' was made by a different version of sphinx. (ie dont think 2.0.6-release would of been available in July)
And somewhere in that version update, the requirement for a .sps file has changed - the new version requires it, whereas the old doesnt. You have no string attributes hence why the file contains no data in the delta.
I would suggest either rebuilding myindex with your current version of indexer - so they versions are identical.
Or maybe you could try copying myindexDelta.sps to myindex.sps. It contains no data (1 dummy byte!) so it shouldn't corrupt anything. Would only need to do this once.

Related

Root or Admin User Not Present in mysql.user (only mysql.infoschema, mysql.session, mysql.sys)

[?] What should I do to normally do sudo mysql -u root -p if it always returns Access denied? [MySQL 8.0] [Ubuntu 20.04]
After clean uninstall mysql and mariadb, then install mysql, at first I got some errors but I can fix it (by configuring my.cnf file and do mysqld --upgrade=force), then finally the mysql service can be started.
However, I got this error whenever tried to access mysql root user
$ sudo mysql -u root -p
Enter password:
ERROR 1045 (28000): Access denied for user 'root'#'localhost' (using password: NO)
Please, I really need help for this :(
And please elaborate me about what I should do to complete the given steps, I'm afraid I don't really understand the basic of Ubuntu system :(
Thank you so much for any effort you guys tried to help me, I really really appreciate it!
What I Tried
To at least access mysql, I add skip-grant-tables on my.cnf.
(However, I cannot do anything such as create user, etc, both inside and outside safe configuration mode.)
$ sudo mysql -u root -p # this only uses blank password
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 8.0.31-0ubuntu0.20.04.2 (Ubuntu)
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SELECT USER();
+--------+
| USER() |
+--------+
| root# |
+--------+
1 row in set (0,01 sec)
mysql> CREATE USER 'root'#'localhost' IDENTIFIED BY 'root';
ERROR 1290 (HY000): The MySQL server is running with the --skip-grant-tables option so it cannot execute this statement
mysql> SELECT user, host FROM mysql.user;
+------------------+-----------+
| user | host |
+------------------+-----------+
| mysql.infoschema | localhost |
| mysql.session | localhost |
| mysql.sys | localhost |
+------------------+-----------+
3 rows in set (0,00 sec)
mysql> SELECT * FROM mysql.user\G
*************************** 1. row ***************************
Host: localhost
User: mysql.infoschema
Select_priv: Y
Insert_priv: N
Update_priv: N
Delete_priv: N
Create_priv: N
Drop_priv: N
Reload_priv: N
Shutdown_priv: N
Process_priv: N
File_priv: N
Grant_priv: N
References_priv: N
Index_priv: N
Alter_priv: N
Show_db_priv: N
Super_priv: N
Create_tmp_table_priv: N
Lock_tables_priv: N
Execute_priv: N
Repl_slave_priv: N
Repl_client_priv: N
Create_view_priv: N
Show_view_priv: N
Create_routine_priv: N
Alter_routine_priv: N
Create_user_priv: N
Event_priv: N
Trigger_priv: N
Create_tablespace_priv: N
...
*************************** 2. row ***************************
Host: localhost
User: mysql.session
Select_priv: N
Insert_priv: N
Update_priv: N
Delete_priv: N
Create_priv: N
Drop_priv: N
Reload_priv: N
Shutdown_priv: Y
Process_priv: N
File_priv: N
Grant_priv: N
References_priv: N
Index_priv: N
Alter_priv: N
Show_db_priv: N
Super_priv: Y
Create_tmp_table_priv: N
Lock_tables_priv: N
Execute_priv: N
Repl_slave_priv: N
Repl_client_priv: N
Create_view_priv: N
Show_view_priv: N
Create_routine_priv: N
Alter_routine_priv: N
Create_user_priv: N
Event_priv: N
Trigger_priv: N
Create_tablespace_priv: N
...
*************************** 3. row ***************************
Host: localhost
User: mysql.sys
Select_priv: N
Insert_priv: N
Update_priv: N
Delete_priv: N
Create_priv: N
Drop_priv: N
Reload_priv: N
Shutdown_priv: N
Process_priv: N
File_priv: N
Grant_priv: N
References_priv: N
Index_priv: N
Alter_priv: N
Show_db_priv: N
Super_priv: N
Create_tmp_table_priv: N
Lock_tables_priv: N
Execute_priv: N
Repl_slave_priv: N
Repl_client_priv: N
Create_view_priv: N
Show_view_priv: N
Create_routine_priv: N
Alter_routine_priv: N
Create_user_priv: N
Event_priv: N
Trigger_priv: N
Create_tablespace_priv: N
...
3 rows in set (0,00 sec)
Also error on doing secure installation
$ mysql_secure_installation
Securing the MySQL server deployment.
Enter password for user root:
Error: Access denied for user 'root'#'localhost' (using password: YES)
my.cnf file
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
# skip-grant-tables
bind-address = 127.0.0.1
max_connections = 100
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
myisam_recover_options = BACKUP
key_buffer_size = 128M
table_open_cache = 400
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
# general_log_file = /var/log/mysql/mysql.log
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
log_bin = /var/log/mysql/mariadb-bin
log_bin_index = /var/log/mysql/mariadb-bin.index
default_storage_engine = InnoDB
innodb_buffer_pool_size = 256M
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
[galera]
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
# Error handling after installation by commenting this out
# system status error handling (these variables unknown)
# query_cache_limit = 128K
# query_cache_size = 64M
# expire_logs_days = 10
# max_binlog_size = 100M
# $ mysqld --upgrade=force error handling (these variables unknown)
# log_warnings = 2
# log_slow_verbosity = query_plan
MySQL status
$ sudo systemctl status mysql
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-12-01 14:40:12 WIB; 12min ago
Main PID: 247973 (mysqld)
Status: "Server is operational"
Tasks: 38 (limit: 6878)
Memory: 312.6M
CGroup: /system.slice/mysql.service
└─247973 /usr/sbin/mysqld
Des 01 14:40:04 EMP10 systemd[1]: Starting MySQL Community Server...
Des 01 14:40:05 EMP10 mysqld[247973]: 2022-12-01T07:40:05.235618Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.31-0ubuntu0.20.04.2) starting as process 247973
Des 01 14:40:05 EMP10 mysqld[247973]: 2022-12-01T07:40:05.272065Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
Des 01 14:40:09 EMP10 mysqld[247973]: 2022-12-01T07:40:09.042423Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
Des 01 14:40:12 EMP10 mysqld[247973]: 2022-12-01T07:40:12.258475Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
Des 01 14:40:12 EMP10 mysqld[247973]: 2022-12-01T07:40:12.258535Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
Des 01 14:40:12 EMP10 mysqld[247973]: 2022-12-01T07:40:12.474475Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
Des 01 14:40:12 EMP10 mysqld[247973]: 2022-12-01T07:40:12.474611Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.31-0ubuntu0.20.04.2' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
Des 01 14:40:12 EMP10 systemd[1]: Started MySQL Community Server.
Actually, if the suggestion is only to uninstall and install mysql, the later installing always gives me errors to handle :(
I usually did these to clean uninstall and install mysql
$ sudo systemctl stop mysql
$ dpkg -l | grep mysql
ii mysql-client-8.0 8.0.31-0ubuntu0.20.04.2 amd64 MySQL database client binaries
ii mysql-client-core-8.0 8.0.31-0ubuntu0.20.04.2 amd64 MySQL database core client binaries
ii mysql-common 1:10.3.37+maria~ubu1804 all MariaDB database common files (e.g. /etc/mysql/my.cnf)
ii mysql-server 8.0.31-0ubuntu0.20.04.2 all MySQL database server (metapackage depending on the latest version)
ii mysql-server-8.0 8.0.31-0ubuntu0.20.04.2 amd64 MySQL database server binaries and system database setup
ii mysql-server-core-8.0 8.0.31-0ubuntu0.20.04.2 amd64 MySQL database server binaries
$ sudo apt remove --purge mysql-client-* mysql-common mysql-server*
$ sudo rm -rf /etc/mysql /var/lib/mysql /var/log/mysql
$ sudo apt autoremove
$ sudo apt autoclean
$ sudo apt install mysql-server # or $ sudo apt install mysql-server-8.0 mysql-server-core-8.0
After installation, there must be something errors to handle.
Reinstalling also didn't fix
$ sudo apt install --reinstall mysql-server # or $ sudo apt install mysql-server-8.0 mysql-server-core-8.0

Moving Prestashop 1.5.6 website from one hosting account to another from the same company now consumes over 25% of the server resources?

I had a Prestashop 1.5.6 website working fine in one hosting account. After moving it to another of supposedly the same features, I started to receive warning it consumes over 25% of the server resources. I know the new account has a different mysql version, I dont know if the problem could be related to that.
The warning messages I got from the hosting provider are
CPU_TIME:606 table_rows_read:357770884 SELECTS:80 ROWS_UPDATED:0 ROWS_FETCHED:442135 BUSY_TIME:668 ONNECTED_TIME:673 BYTES_SENT:27099136 BYTES_RECEIVED:17676 WAIT_TIME:62
Top table row reads:
DB_USER: ********** -- TOTAL_CONNECTIONS: 4 -- CONNECTED_TIME: 673 -- CPU_TIME: 606 -- TABLE_ROW_READS: 357770884 -- SELECT_COMMANDS: 80 -- UPDATE_COMMANDS: -- BUSY_TIME: 668 -- BYTES_SENT: 27099136 -- BYTES_RECEIVED: 17676 -- WAIT_TIME (IO): 62
Top WAIT (IO) TIME:
DB_USER: ********** -- TOTAL_CONNECTIONS: 4 -- CONNECTED_TIME: 673 -- CPU_TIME: 606 -- TABLE_ROW_READS: 357770884 -- SELECT_COMMANDS: 80 -- UPDATE_COMMANDS: -- BUSY_TIME: 668 -- BYTES_SENT: 27099136 -- BYTES_RECEIVED: 17676 -- WAIT_TIME (IO): 62
SN 03:31 0:01 /index.php
SN 03:32 0:02 /index.php
SN 03:34 0:01 /index.php
SN 03:35 0:00 /index.php
Fri Feb 23 03:35:58 CST 2018
Running Processes:
S 03:08 0:00 dovecot/imap
SN 03:31 0:01 /index.php
SN 03:32 0:02 /index.php
SN 03:34 0:01 /index.php
SN 03:35 0:00 /index.php
Running Queries:
*************************** 1. row ***************************
USER: *********
DB: *********
STATE: Sending data
TIME: 34
COMMAND: Query
INFO: SELECT c.`name`, cl.`id_lang`, IF(cl.`id_lang` IS NULL, c.`value`, cl.`value`) AS value, c.id_shop_group, c.id_shop
FROM `ps_configuration` c
LEFT JOIN `ps_configuration_lang` cl ON (c.id_configuration = cl.id_configuration)
*************************** 2. row ***************************
USER: **********
DB: ***********
STATE: Copying to tmp table
TIME: 10
COMMAND: Query
INFO: SELECT h.id_hook, h.name as h_name, title, description, h.position, live_edit, hm.position as hm_position, m.id_module, m.name, active
FROM `ps_hook` h
INNER JOIN `ps_hook_module` hm ON (h.id_hook = hm.id_hook AND hm.id_shop = 1)
INNER JOIN `ps_module` as m ON (m.id_module = hm.id_module)
ORDER BY hm.position
*************************** 3. row ***************************
USER: **********
DB: **********
STATE: Copying to tmp table
TIME: 102
COMMAND: Query
INFO: SELECT h.id_hook, h.name as h_name, title, description, h.position, live_edit, hm.position as hm_position, m.id_module, m.name, active
FROM `ps_hook` h
INNER JOIN `ps_hook_module` hm ON (h.id_hook = hm.id_hook AND hm.id_shop = 1)
INNER JOIN `ps_module` as m ON (m.id_module = hm.id_module)
ORDER BY hm.position
*************************** 4. row ***************************
USER: **********
DB: **********
STATE: Copying to tmp table
TIME: 178
COMMAND: Query
INFO: SELECT h.id_hook, h.name as h_name, title, description, h.position, live_edit, hm.position as hm_position, m.id_module, m.name, active
FROM `ps_hook` h
INNER JOIN `ps_hook_module` hm ON (h.id_hook = hm.id_hook AND hm.id_shop = 1)
INNER JOIN `ps_module` as m ON (m.id_module = hm.id_module)
ORDER BY hm.position
Open connections
Current Site Requests:
index.php?id_category=20&controller=category&id_la
index.php?id_product=212&controller=product&id_lan
index.php?id_product=212&controller=product&id_lan
index.php?id_product=247&controller=product&id_lan
index.php?id_product=88&controller=product&id_lang
The table structures are
Did you try profiling ? There is a great tool inside Prestashop that you can activate by modifying config/defines.inc.php :
define('_PS_DEBUG_PROFILING_', true);
I also would suggest that you take a look at PHP Versions, as it seems like their would be some major change in how scripts are handled for example between 5.2 and 7.1 versions.

Preventing MySQL From Crashing? “Creeping” Memory Leak [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
I set up a fresh 1GB VPS server instance dedicated solely to MySQL. Everything seems to be working great. However, I noticed that mysqld memory usage quickly grows and sort of peaks out at about 700MB (as expected), but then it slowly “creeps” up over the course of 1-2 days. Then when it reaches about 770MB, the process gets killed by OOM Killer and restarts within a few seconds. It’s not a massive downtime, but I would like it to be stable.
I am using MySQL version 5.7.21. Here are the variables that I changed from default in the my.cnf file, everything else is set to the defaults. The biggest change is an increase memory to the innodb_buffer_pool_size to 512M:
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
[mysqld]
sql_mode = "NO_ENGINE_SUBSTITUTION"
innodb_buffer_pool_size = 384M
innodb_log_buffer_size = 2097152
innodb_log_file_size = 20971520
innodb_strict_mode = OFF
join_buffer_size = 1048576
key_buffer_size = 88080384
max_connect_errors = 10000
max_connections = 151
myisam_recover_options = "BACKUP,FORCE"
performance_schema = 0
read_buffer_size = 1048576
slow_query_log = ON
sort_buffer_size = 1048576
sync_binlog = 0
thread_stack = 262144
wait_timeout = 14400
I’m kind of noobish with MySQL administration, so I’m hoping someone with more experience can provide some advice to keeping my MySQL instance a bit more stable on my 1GB instance, OOM Killer strategies and making the database faster/efficient at the same time.
EDIT: I added some extra files for additional information:
SHOW GLOBAL STATUS: https://pastebin.com/SSVEJrQc
SHOW GLOBAL VARIABLES: https://pastebin.com/gV5yGdFR
SHOW ENGINE INNODB STATUS: https://pastebin.com/suHwbpiP
/var/log/mysql/error.log : https://pastebin.com/cRFGrNTp (combined past couple days)
The my.cnf file is shown at the top. I made some changes to it to reduce some of the memory usage, such as dropping the innodb_buffer_pool_size down to 384M instead of 512M. According to "top -c", the memory usage quickly rose to around 550MB and slowly creeped up to about 740M, no crash yet.
Also, the server instance has 1GB of RAM, so I'm not sure why MySQL has to crash at around 770M. It's just a fresh install of Ubuntu 16.04 and MySQL, nothing else at all, no apache or php.
EDIT: I've included more data, I'm running instance on a Dreamhost DreamCompute cloud server:
"top" Results: https://pastebin.com/RNBYMf0b
"df -h" results:
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
tmpfs 100M 11M 89M 11% /run
/dev/vda1 78G 22G 56G 29% /
tmpfs 497M 0 497M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 497M 0 497M 0% /sys/fs/cgroup
tmpfs 100M 0 100M 0% /run/user/1000
"iostat -x": https://pastebin.com/xUNu7fEi
"mysqltuner": https://pastebin.com/pVC3kN1C
I'm seeing a lot of "tmpfs" directories (using RAM)? Is there anything that should or shouldn't be on this list? Also, I've just now updated the my.cnf with the values Wilson Hauck provided and will restart mysqld.
Edit: Regarding "ulimit", after reading more about it, I edited /etc/security/limits.conf and added the following: "* - nofile 40000" . Now when I use "sudo sh -c "ulimit -n", it shows 40000 instead of 1024.
The "iostat -x" and "mysqltuner" reports were created right after I restarted MySQL because they wouldn't install due to "not enough memory". Thanks Wilson for the mysqltuner suggestion, that looks like a great program.
UPDATE: 2/25/2018:
With all of the recommended settings set, I had mysql again crash about 10 hours ago. Here are the details from syslog:
Feb 25 09:41:52 database kernel: [739566.195215] snapd invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=-900
Feb 25 09:41:52 database kernel: [739566.195276] snapd cpuset=/ mems_allowed=0
Feb 25 09:41:52 database kernel: [739566.195310] CPU: 0 PID: 1228 Comm: snapd Not tainted 4.4.0-112-generic #135-Ubuntu
Feb 25 09:41:52 database kernel: [739566.195311] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
Feb 25 09:41:52 database kernel: [739566.195313] 0000000000000286 245c0a16fc0af5b0 ffff8800000839d8 ffffffff813fc233
Feb 25 09:41:52 database kernel: [739566.195317] ffff880000083b90 ffff88003ae39c00 ffff880000083a48 ffffffff8120dafe
Feb 25 09:41:52 database kernel: [739566.195319] ffffffff81cd8367 0000000000000000 ffffffff81e6b1a0 0000000000000206
Feb 25 09:41:52 database kernel: [739566.195321] Call Trace:
Feb 25 09:41:52 database kernel: [739566.195361] [<ffffffff813fc233>] dump_stack+0x63/0x90
Feb 25 09:41:52 database kernel: [739566.195375] [<ffffffff8120dafe>] dump_header+0x5a/0x1c5
Feb 25 09:41:52 database kernel: [739566.195383] [<ffffffff811946a2>] oom_kill_process+0x202/0x3c0
Feb 25 09:41:52 database kernel: [739566.195385] [<ffffffff81194ac9>] out_of_memory+0x219/0x460
Feb 25 09:41:52 database kernel: [739566.195394] [<ffffffff8119aad5>] __alloc_pages_slowpath.constprop.88+0x965/0xb00
Feb 25 09:41:52 database kernel: [739566.195396] [<ffffffff8119aef6>] __alloc_pages_nodemask+0x286/0x2a0
Feb 25 09:41:52 database kernel: [739566.195404] [<ffffffff811e483c>] alloc_pages_current+0x8c/0x110
Feb 25 09:41:52 database kernel: [739566.195406] [<ffffffff81190c6b>] __page_cache_alloc+0xab/0xc0
Feb 25 09:41:52 database kernel: [739566.195407] [<ffffffff8119317a>] filemap_fault+0x14a/0x3f0
Feb 25 09:41:52 database kernel: [739566.195418] [<ffffffff812a5d56>] ext4_filemap_fault+0x36/0x50
Feb 25 09:41:52 database kernel: [739566.195419] [<ffffffff811bfe70>] __do_fault+0x50/0xe0
Feb 25 09:41:52 database kernel: [739566.195421] [<ffffffff811c39c2>] handle_mm_fault+0xfa2/0x1820
Feb 25 09:41:52 database kernel: [739566.195433] [<ffffffff810bbc6c>] ? set_next_entity+0x9c/0xb0
Feb 25 09:41:52 database kernel: [739566.195443] [<ffffffff8106b687>] __do_page_fault+0x197/0x400
Feb 25 09:41:52 database kernel: [739566.195445] [<ffffffff8106b957>] trace_do_page_fault+0x37/0xe0
Feb 25 09:41:52 database kernel: [739566.195450] [<ffffffff81063f29>] do_async_page_fault+0x19/0x70
Feb 25 09:41:52 database kernel: [739566.195464] [<ffffffff81849af8>] async_page_fault+0x28/0x30
Feb 25 09:41:52 database kernel: [739566.195465] Mem-Info:
Feb 25 09:41:52 database kernel: [739566.195470] active_anon:220242 inactive_anon:1399 isolated_anon:0
Feb 25 09:41:52 database kernel: [739566.195470] active_file:799 inactive_file:1712 isolated_file:0
Feb 25 09:41:52 database kernel: [739566.195470] unevictable:913 dirty:1 writeback:0 unstable:0
Feb 25 09:41:52 database kernel: [739566.195470] slab_reclaimable:5664 slab_unreclaimable:3906
Feb 25 09:41:52 database kernel: [739566.195470] mapped:1931 shmem:2691 pagetables:1544 bounce:0
Feb 25 09:41:52 database kernel: [739566.195470] free:12712 free_pcp:113 free_cma:0
Feb 25 09:41:52 database kernel: [739566.195473] Node 0 DMA free:4548kB min:716kB low:892kB high:1072kB active_anon:5548kB inactive_anon:12kB active_file:1068kB inactive_file:1968kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:4kB writeback:0kB mapped:808kB shmem:364kB slab_reclaimable:220kB slab_unreclaimable:500kB kernel_stack:368kB pagetables:936kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:19320 all_unreclaimable? yes
Feb 25 09:41:52 database kernel: [739566.195491] lowmem_reserve[]: 0 958 958 958 958
Feb 25 09:41:52 database kernel: [739566.195495] Node 0 DMA32 free:46300kB min:44336kB low:55420kB high:66504kB active_anon:875420kB inactive_anon:5584kB active_file:2128kB inactive_file:4880kB unevictable:3652kB isolated(anon):0kB isolated(file):0kB present:1032184kB managed:1000192kB mlocked:3652kB dirty:0kB writeback:0kB mapped:6916kB shmem:10400kB slab_reclaimable:22436kB slab_unreclaimable:15124kB kernel_stack:3376kB pagetables:5240kB unstable:0kB bounce:0kB free_pcp:452kB local_pcp:452kB free_cma:0kB writeback_tmp:0kB pages_scanned:42084 all_unreclaimable? yes
Feb 25 09:41:52 database kernel: [739566.195502] lowmem_reserve[]: 0 0 0 0 0
Feb 25 09:41:52 database kernel: [739566.195504] Node 0 DMA: 1*4kB (U) 6*8kB (UME) 81*16kB (UME) 32*32kB (UM) 8*64kB (ME) 7*128kB (ME) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4548kB
Feb 25 09:41:52 database kernel: [739566.195514] Node 0 DMA32: 107*4kB (MEH) 240*8kB (ME) 249*16kB (UMEH) 141*32kB (UMEH) 108*64kB (UMEH) 59*128kB (UMEH) 20*256kB (UMEH) 13*512kB (UME) 9*1024kB (UMH) 0*2048kB 0*4096kB = 46300kB
Feb 25 09:41:52 database kernel: [739566.195526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 25 09:41:52 database kernel: [739566.195560] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 25 09:41:52 database kernel: [739566.195561] 5817 total pagecache pages
Feb 25 09:41:52 database kernel: [739566.195576] 0 pages in swap cache
Feb 25 09:41:52 database kernel: [739566.195581] Swap cache stats: add 0, delete 0, find 0/0
Feb 25 09:41:52 database kernel: [739566.195582] Free swap = 0kB
Feb 25 09:41:52 database kernel: [739566.195583] Total swap = 0kB
Feb 25 09:41:52 database kernel: [739566.195584] 262044 pages RAM
Feb 25 09:41:52 database kernel: [739566.195585] 0 pages HighMem/MovableOnly
Feb 25 09:41:52 database kernel: [739566.195585] 8019 pages reserved
Feb 25 09:41:52 database kernel: [739566.195586] 0 pages cma reserved
Feb 25 09:41:52 database kernel: [739566.195587] 0 pages hwpoisoned
Feb 25 09:41:52 database kernel: [739566.195588] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 25 09:41:52 database kernel: [739566.195593] [ 348] 0 348 9237 1199 21 3 0 0 systemd-journal
Feb 25 09:41:52 database kernel: [739566.195595] [ 451] 0 451 25742 46 17 3 0 0 lvmetad
Feb 25 09:41:52 database kernel: [739566.195597] [ 452] 0 452 10744 376 24 3 0 -1000 systemd-udevd
Feb 25 09:41:52 database kernel: [739566.195599] [ 550] 100 550 25081 61 19 3 0 0 systemd-timesyn
Feb 25 09:41:52 database kernel: [739566.195601] [ 940] 0 940 4030 222 11 3 0 0 dhclient
Feb 25 09:41:52 database kernel: [739566.195603] [ 1054] 0 1054 1305 29 8 3 0 0 iscsid
Feb 25 09:41:52 database kernel: [739566.195604] [ 1055] 0 1055 1430 877 8 3 0 -17 iscsid
Feb 25 09:41:52 database kernel: [739566.195606] [ 1063] 0 1063 68647 1056 37 3 0 0 accounts-daemon
Feb 25 09:41:52 database kernel: [739566.195608] [ 1072] 0 1072 6932 491 19 3 0 0 cron
Feb 25 09:41:52 database kernel: [739566.195609] [ 1078] 0 1078 6511 337 18 3 0 0 atd
Feb 25 09:41:52 database kernel: [739566.195611] [ 1091] 0 1091 1099 300 8 3 0 0 acpid
Feb 25 09:41:52 database kernel: [739566.195613] [ 1094] 0 1094 7136 99 19 3 0 0 systemd-logind
Feb 25 09:41:52 database kernel: [739566.195614] [ 1099] 0 1099 16377 273 35 3 0 -1000 sshd
Feb 25 09:41:52 database kernel: [739566.195616] [ 1101] 104 1101 64098 348 27 3 0 0 rsyslogd
Feb 25 09:41:52 database kernel: [739566.195618] [ 1105] 107 1105 10722 380 27 3 0 -900 dbus-daemon
Feb 25 09:41:52 database kernel: [739566.195619] [ 1113] 0 1113 70365 2590 31 6 0 -900 snapd
Feb 25 09:41:52 database kernel: [739566.195621] [ 1114] 0 1114 158952 1017 31 4 0 0 lxcfs
Feb 25 09:41:52 database kernel: [739566.195623] [ 1150] 0 1150 3343 36 11 3 0 0 mdadm
Feb 25 09:41:52 database kernel: [739566.195624] [ 1159] 0 1159 69294 181 38 3 0 0 polkitd
Feb 25 09:41:52 database kernel: [739566.195626] [ 1203] 0 1203 3618 374 12 3 0 0 agetty
Feb 25 09:41:52 database kernel: [739566.195627] [13137] 0 13137 3664 356 11 3 0 0 agetty
Feb 25 09:41:52 database kernel: [739566.195629] [13141] 0 13141 3664 329 12 3 0 0 agetty
Feb 25 09:41:52 database kernel: [739566.195631] [ 4475] 112 4475 345228 190282 441 4 0 0 mysqld
Feb 25 09:41:52 database kernel: [739566.195635] [28991] 0 28991 1126 141 8 3 0 0 apt.systemd.dai
Feb 25 09:41:52 database kernel: [739566.195638] [28998] 0 28998 1126 383 8 3 0 0 apt.systemd.dai
Feb 25 09:41:52 database kernel: [739566.195640] [29036] 0 29036 11324 1282 27 3 0 0 apt-get
Feb 25 09:41:52 database kernel: [739566.195641] [29217] 0 29217 11324 867 23 3 0 0 apt-get
Feb 25 09:41:52 database kernel: [739566.195643] [29220] 0 29220 1126 157 8 3 0 0 sh
Feb 25 09:41:52 database kernel: [739566.195645] [29221] 0 29221 1126 367 9 3 0 0 update-motd-upd
Feb 25 09:41:52 database kernel: [739566.195646] [29235] 0 29235 40967 16609 84 3 0 0 apt-check
Feb 25 09:41:52 database kernel: [739566.195648] [29570] 0 29570 12235 358 28 3 0 0 cron
Feb 25 09:41:52 database kernel: [739566.195649] [29571] 0 29571 1126 144 8 3 0 0 sh
Feb 25 09:41:52 database kernel: [739566.195651] [29572] 0 29572 2809 278 10 3 0 0 bash
Feb 25 09:41:52 database kernel: [739566.195653] [29579] 0 29579 12235 290 28 3 0 0 cron
Feb 25 09:41:52 database kernel: [739566.195654] [29580] 0 29580 1126 163 7 3 0 0 sh
Feb 25 09:41:52 database kernel: [739566.195656] [29581] 0 29581 2809 91 9 3 0 0 bash
Feb 25 09:41:52 database kernel: [739566.195658] [29582] 0 29582 14775 135 30 3 0 0 sshd
Feb 25 09:41:52 database kernel: [739566.195659] [29584] 0 29584 12235 358 28 3 0 0 cron
Feb 25 09:41:52 database kernel: [739566.195661] [29589] 0 29589 14775 134 32 3 0 0 sshd
Feb 25 09:41:52 database kernel: [739566.195662] [29592] 0 29592 12855 431 30 3 0 0 sudo
Feb 25 09:41:52 database kernel: [739566.195664] [29597] 0 29597 1126 70 8 3 0 0 sh
Feb 25 09:41:52 database kernel: [739566.195665] [29598] 0 29598 2809 124 9 3 0 0 bash
Feb 25 09:41:52 database kernel: [739566.195667] [29602] 0 29602 12235 357 28 3 0 0 cron
Feb 25 09:41:52 database kernel: [739566.195668] [29603] 0 29603 1126 92 8 3 0 0 sh
Feb 25 09:41:52 database kernel: [739566.195670] [29604] 0 29604 2809 321 10 3 0 0 bash
Feb 25 09:41:52 database kernel: [739566.195672] [29615] 0 29615 5787 36 13 3 0 0 systemctl
Feb 25 09:41:52 database kernel: [739566.195673] [29620] 0 29620 12235 276 28 3 0 0 cron
Feb 25 09:41:52 database kernel: [739566.195675] [29621] 0 29621 11236 413 26 3 0 0 sudo
Feb 25 09:41:52 database kernel: [739566.195677] [29623] 0 29623 1126 139 8 3 0 0 sh
Feb 25 09:41:52 database kernel: [739566.195678] [29624] 0 29624 2807 234 10 3 0 0 bash
Feb 25 09:41:52 database kernel: [739566.195680] [29628] 0 29628 14775 75 30 3 0 0 sshd
Feb 25 09:41:52 database kernel: [739566.195681] [29629] 0 29629 12235 476 28 3 0 0 cron
Feb 25 09:41:52 database kernel: [739566.195683] [29641] 0 29641 6945 111 18 3 0 0 sudo
Feb 25 09:41:52 database kernel: [739566.195684] [29642] 0 29642 6945 87 18 3 0 0 sudo
Feb 25 09:41:52 database kernel: [739566.195686] [29643] 0 29643 1126 152 8 3 0 0 sh
Feb 25 09:41:52 database kernel: [739566.195687] [29644] 0 29644 345 1 5 3 0 0 bash
Feb 25 09:41:52 database kernel: [739566.195689] Out of memory: Kill process 4475 (mysqld) score 750 or sacrifice child
Feb 25 09:41:52 database kernel: [739566.197374] Killed process 4475 (mysqld) total-vm:1380912kB, anon-rss:761128kB, file-rss:0kB
Feb 25 09:41:53 database kernel: [739566.658202] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20807 DF PROTO=TCP SPT=54168 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739566.658214] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20808 DF PROTO=TCP SPT=49412 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739566.658291] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20809 DF PROTO=TCP SPT=54293 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739566.666146] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20810 DF PROTO=TCP SPT=53588 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739566.666160] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20811 DF PROTO=TCP SPT=53612 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database systemd[1]: Started MySQL Community Server.
Feb 25 09:41:53 database CRON[29570]: (CRON) info (No MTA installed, discarding output)
Feb 25 09:41:53 database systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
Feb 25 09:41:53 database systemd[1]: mysql.service: Unit entered failed state.
Feb 25 09:41:53 database systemd[1]: mysql.service: Failed with result 'signal'.
Feb 25 09:41:53 database systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Feb 25 09:41:53 database systemd[1]: Stopped MySQL Community Server.
Feb 25 09:41:53 database systemd[1]: Starting MySQL Community Server...
Feb 25 09:41:53 database kernel: [739567.046536] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=42 ID=21475 DF PROTO=TCP SPT=37118 DPT=22 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739567.066187] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20869 DF PROTO=TCP SPT=54293 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739567.066313] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20870 DF PROTO=TCP SPT=49412 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739567.066324] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.*** DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20871 DF PROTO=TCP SPT=54168 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:53 database kernel: [739567.082221] [UFW BLOCK] IN=ens3 OUT= MAC=fa:16:3e:bc:28:e3:44:f4:77:a7:c0:20:08:00 SRC=***.***.***.***DST=***.***.***.*** LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=20873 DF PROTO=TCP SPT=53612 DPT=3306 WINDOW=0 RES=0x00 RST URGP=0
Feb 25 09:41:54 database kernel: [739567.524532] audit: type=1400 audit(1519551714.232:296): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/29710/status" pid=29710 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Feb 25 09:41:54 database kernel: [739567.524603] audit: type=1400 audit(1519551714.232:297): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=29710 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
Feb 25 09:41:54 database kernel: [739567.524670] audit: type=1400 audit(1519551714.232:298): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/29710/status" pid=29710 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Feb 25 09:41:56 database systemd[1]: Started MySQL Community Server.
According to MySQLtuner, it says: "Maximum possible memory usage: 664.8M (66.99% of installed RAM)", so it must be exceeding this somehow.
Suggestions to consider for your my.cnf/ini [mysqld] section,
ulimit -n 40000 # at your Linux command prompt to raise n open files limit
table_open_cache=10000 # from 2000 to support 1M+ opened in 2 days
table_definition_cache=2500 # from default to support 2000+ opened in 2 days
open_files_limit=30000 # from 5000 to support 900,000 + opened in 2 days
max_connections=50 # from 151 to support 17 max_used_connections
read_rnd_buffer_size=128K # from 256k default to reduce RD RPS
innodb_change_buffer_max_size=15 # from 25% of innodb_buffer_pool_size 1% used
innodb_log_buffer_size=12M # from 2M to cover 30 minutes of log
innodb_log_file_size=120M # from ~ 20M to cover a few days
#max_allowed_packet=16M # lead with # for default of 1M m_a_p
if you need more than 1M, in your SESSION
SET #max_allowed_packet=nnnnnnnn; up to 1G and 1G is the LIMIT.
query_cache_size=0 # from 16M - it is already OFF, do not waste RAM on it
query_cache_limit=1K # from 1M to conserve more RAM
query_cache_min_res_unit=512 # from 4096 to store more small results, if ever used
innodb_buffer_pool_instances=8 # from 1 to minimize mutex contention
will be fine with you UNDER 1G per instance of innodb_buffer_pool_size
innodb_lru_scan_depth=128 # from 1024 which is causing page_cleaner warnings
innodb_page_cleaners=64 # from 1 to auto follow = innodb_buffer_pool_instances
thread_cache_size=50 # from 8 default to support 17 max_used with room for growth.
from my perspective, backup your current my.cnf/ini and implement all. There are more opportunities for another day.
Could we use Stack Overflow's chat next week ?
Your query_cache_size is 52428800 = 50M
innodb_buffer_pool_size = 512M
key_buffer_size = 88080384 = 84M
That's already ~650M, which leaves MySQL about 50M for 'other stuff'.
If you tweak down some of these settings, chances are that you can stay below 700M. Unless your version of MySQL really has a memory leak.
Try reducing some of these numbers by another 50M in total.
Bonus:
Just found this calculator: http://www.mysqlcalculator.com/. Might help you figure this out more accurately. Can't speak to how accurate this tool is though
To answer part of your question:
Also, the server instance has 1GB of RAM, so I'm not sure why MySQL has to crash at around 770M. It's just a fresh install of Ubuntu 16.04 and MySQL, nothing else at all, no apache or php.
A server doesn't have "nothing else at all" even if you don't install anything else yourself, there's always background stuff going on as well, and whatever your VPS image includes. You can use top (or if available my preferred tool, htop) to see what's running and what's using memory.
In the OOM kill log, for instance, apt-check has an rss size of 16848, meaning it's taking another 70GB of RAM by itself (see this answer on OOM kiler logs - the numbers indicate 4kB blocks). Additionally, many of the system "directories" such as /tmp are actually stored in RAM rather than on disk. You can see if that's the caes on your machine by running df -h - anything listed with tmpfs as its filesystem is stored in RAM, and if it has space being used, is using RAM as well.
Put various things together, and it's plausible that various system processes can take a decent chunk of that RAM overhead, even without any other processes you specifically installed.

How to write a set of fields to JSON?

I am trying to write few fields in my data frame to JSON. My data structure in the data frame is
Key|col1|col2|col3|col4
key|a |b |c |d
Key|a1 |b1 |c1 |d1
Now i am trying to convert just the col1 to col4 fields to JSON and give a name to the Json field
expected output
[Key,{cols:[{col1:a,col2:b,col3:c,col4:d},{col1:a1,col2:b1,col3:c1,col4:d1}]
I wrote a udf for this.
val summary = udf(
(col1:String, col2:String, col3:String, col4:String) => "{\"cols\":[" + " {\"col1\":" + col1 + ",\"col2\":" + col2 + ",\"col3\":" + col3 + ",\"col4\":" + col4 + "}]}"
)
val result = input.withColumn("Summary",summary('col1,'col2,'col3,'col4))
val result1 = result.select('Key,'Summary)
result1.show(10)
This is my result
[Key,{cols:[{col1:a,col2:b,col3:c,col4:d}]}]
[Key,{cols:[{col1:a1,col2:b1,col3:c1,col4:d1}]}]
As you can see, they are not grouped. Is there a way to group these rows using the UDF itself. I am new to scala/Spark and not able to figure out the proper udf.
// Create your dataset
scala> val ds = Seq((1, "hello", 1L), (2, "world", 2L)).toDF("id", "token", "long")
ds: org.apache.spark.sql.DataFrame = [id: int, token: string ... 1 more field]
// select the fields you want to map to json
scala> ds.select('token, 'long).write.json("your-json")
// check the result
➜ spark git:(master) ✗ ls -ltr your-json/
total 16
-rw-r--r-- 1 jacek staff 27 11 kwi 17:18 part-r-00007-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 0 11 kwi 17:18 part-r-00006-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 0 11 kwi 17:18 part-r-00005-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 0 11 kwi 17:18 part-r-00004-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 27 11 kwi 17:18 part-r-00003-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 0 11 kwi 17:18 part-r-00002-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 0 11 kwi 17:18 part-r-00001-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 0 11 kwi 17:18 part-r-00000-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
-rw-r--r-- 1 jacek staff 0 11 kwi 17:18 _SUCCESS
➜ spark git:(master) ✗ cat your-json/part-r-00003-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
{"token":"hello","long":1}
➜ spark git:(master) ✗ cat your-json/part-r-00007-91f81f62-54bb-42ae-bddc-33829a0e3c16.json
{"token":"world","long":2}
UDFs will map one row to one row. If you have multiple rows in your DataFrame that you want to combine into one element, you're going to need to use a function like reduceByKey that aggregates multiple rows.
There may be a DataFrame specific function to do this, but I would do this processing with the RDD functionality, like so:
val colSummary = udf(
(col1:String, col2:String, col3:String, col4:String) => "{\"col1\":" + col1 + ",\"col2\":" + col2 + ",\"col3\":" + col3 + ",\"col4\":" + col4 + "}"
)
val colRDD = input.withColumn("Summary",summary('col1,'col2,'col3,'col4)).rdd.map(x => (x.getString(0),x.getString(5)))
This gives us a RDD[(String,String)], which will allow us to use the PairRDDFunctions like reduceByKey (see the docs). The key of the tuple is the original key, and the value is the json encoding for a single element that we need to aggregate together to make the cols list. We glue them all together into a comma-separated list, and then we add the beginning and end, and then we're done.
val result = colRDD.reduceByKey((x,y) => (x+","+y)).map(x => "["+x._1+",{\"cols\":["+x._2+"]}]")
result.take(10)

Aliases not working

I'm trying to setup a mailserver with postfix and dovecot, however I can't get aliases to work for some reason. Everytime I send a mail to an alias I get "User unknown" or "User unknown in virtual mailbox table". I've used postmap -q alias#example.com mysql:/etc/postfix/mysql-virtual-alias-maps.cfand it returns user#example.com as expected. I've also enabled query logging in mysql and I can see the queries are executed, but for some reason it still checks if the user alias#example.com exists, which of course it doesn't.
main.cf:
myhostname = xx
smtpd_banner = \$myhostname ESMTP
biff = no
append_dot_mydomain = no
readme_directory = /usr/share/doc/postfix
smtpd_tls_cert_file = /etc/ssl/certs/xx.pem
smtpd_tls_key_file = /etc/ssl/private/xx.key
smtpd_use_tls = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination =
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
html_directory = /usr/share/doc/postfix/html
inet_protocols = ipv4
virtual_transport = dovecot
virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf
virtual_minimum_uid = 100
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
smtpd_helo_restrictions = reject_unknown_helo_hostname
smtpd_sender_restrictions = reject_unknown_sender_domain
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_rbl_client zen.spamhaus.org, reject_rhsbl_reverse_client dbl.spamhaus.org, reject_rhsbl_helo dbl.spamhaus.org, reject_rhsbl_sender dbl.spamhaus.org
smtpd_data_restrictions = reject_unauth_pipelining
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
dovecot_destination_recipient_limit = 1
receive_override_options = no_address_mappings
milter_connect_macros = j {daemon_name} v {if_name} _
milter_default_action = accept
milter_protocol = 2
smtpd_milters = unix:/spamass/spamass.sock, unix:/clamav/clamav-milter.ctl, inet:localhost:12301
non_smtpd_milters = unix:/spamass/spamass.sock, unix:/clamav/clamav-milter.ctl, inet:localhost:12301
mysql-virtual-alias-maps.cf:
user = mailcp
password = xx
hosts = 127.0.0.1
dbname = mailcp
query = SELECT CONCAT(user_alias.name, '#', domain.name) as destination FROM user_alias INNER JOIN domain ON user_alias.domainID = domain.id WHERE user_alias.primary = 1 AND user_alias.userID IN (SELECT user_alias.userID FROM user_alias INNER JOIN domain ON user_alias.domainID = domain.id WHERE CONCAT(user_alias.name, '#', domain.name)='%s');
mysql-virtual-mailbox-domains.cf:
user = mailcp
password = xx
hosts = 127.0.0.1
dbname = mailcp
query = SELECT 1 FROM domain WHERE name='%s';
mysql-virtual-mailbox-maps.cf:
user = mailcp
password = xx
hosts = 127.0.0.1
dbname = mailcp
query = SELECT 1 FROM user_alias INNER JOIN domain ON user_alias.domainID = domain.id WHERE user_alias.primary = 1 AND CONCAT(user_alias.name, '#', domain.name)='%s';
dovecot-sql.conf.ext:
driver = mysql
connect = host=127.0.0.1 dbname=databasename user=username password=changepwd
default_pass_scheme = SHA512-CRYPT
password_query = SELECT CONCAT(user_alias.name, '#', domain.name) as user, user.password as password, concat('*:bytes=', (CASE WHEN user.quota -1 THEN user.quota ELSE domain.quota END)*1000000) AS userdb_quota_rule FROM user INNER JOIN user_alias ON user.id = user_alias.userID AND user_alias.primary = 1 INNER JOIN domain ON user_alias.domainID = domain.id WHERE CONCAT(user_alias.name, '#', domain.name) ='%u' AND user.status = 1
user_query = SELECT CONCAT('/var/mail/vmail/', domain.name, '/', user_alias.name) as home, 5000 AS uid, 5000 AS gid, concat('*:bytes=', (CASE WHEN user.quota != -1 THEN user.quota ELSE domain.quota END)*1000000) AS userdb_quota_rule FROM user INNER JOIN user_alias ON user.id = user_alias.userID AND user_alias.primary = 1 INNER JOIN domain ON user_alias.domainID = domain.id WHERE CONCAT(user_alias.name, '#', domain.name) ='%u' AND user.status = 1
mail.info (for one mail)
Oct 26 16:38:27 xx postfix/smtpd[538]: connect from mail-wg0-f51.google.com[74.125.82.51]
Oct 26 16:38:28 xx postfix/smtpd[538]: 1FD73580A0F: client=mail-wg0-f51.google.com[74.125.82.51]
Oct 26 16:38:28 xx postfix/cleanup[547]: 1FD73580A0F: message-id=<CAGPeO-YPiiA5TerbFqZ5j5t-PYtGuR2wSaKXUN1=ewTdO-0=Zw#mail.gmail.com>
Oct 26 16:38:28 xx spamd[29841]: spamd: connection from localhost.localdomain [127.0.0.1] at port 58956
Oct 26 16:38:28 xx spamd[29841]: spamd: processing message <CAGPeO-YPiiA5TerbFqZ5j5t-PYtGuR2wSaKXUN1=ewTdO-0=Zw#mail.gmail.com> for xx:1001
Oct 26 16:38:33 xx spamd[29841]: spamd: clean message (-0.7/5.0) for xx:1001 in 4.9 seconds, 1789 bytes.
Oct 26 16:38:33 xx spamd[29841]: spamd: result: . 0 - FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_DKIM_INVALID scantime=4.9,size=1789,user=xx,uid=1001,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=58956,mid=<CAGPeO-YPiiA5TerbFqZ5j5t-PYtGuR2wSaKXUN1=ewTdO-0=Zw#mail.gmail.com>,autolearn=ham
Oct 26 16:38:33 xx opendkim[25686]: 1FD73580A0F: mail-wg0-f51.google.com [74.125.82.51] not internal
Oct 26 16:38:33 xx opendkim[25686]: 1FD73580A0F: not authenticated
Oct 26 16:38:33 xx opendkim[25686]: 1FD73580A0F: DKIM verification successful
Oct 26 16:38:33 xx spamd[29840]: prefork: child states: II
Oct 26 16:38:33 xx postfix/qmgr[477]: 1FD73580A0F: from=<xx#gmail.com>, size=1749, nrcpt=1 (queue active)
Oct 26 16:38:33 xx dovecot: auth-worker(552): mysql(127.0.0.1): Connected to database mailcp
Oct 26 16:38:33 xx postfix/pipe[550]: 1FD73580A0F: to=<xx#xx.in>, relay=dovecot, delay=5.1, delays=5/0/0/0.03, dsn=5.1.1, status=bounced (user unknown)
Oct 26 16:38:33 xx postfix/cleanup[547]: 2012B580A53: message-id=<20141026153833.2012B580A53#xx.in>
Oct 26 16:38:33 xx postfix/bounce[554]: 1FD73580A0F: sender non-delivery notification: 2012B580A53
Oct 26 16:38:33 xx postfix/qmgr[477]: 2012B580A53: from=<>, size=3908, nrcpt=1 (queue active)
Oct 26 16:38:33 xx postfix/qmgr[477]: 1FD73580A0F: removed
Oct 26 16:38:33 xx postfix/smtpd[538]: disconnect from mail-wg0-f51.google.com[74.125.82.51]
Oct 26 16:38:33 xx postfix/smtp[556]: 2012B580A53: to=<xx#gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.195.27]:25, delay=0.2, delays=0/0/0.09/0.1, dsn=2.0.0, status=sent (250 2.0.0 OK 1414337913 dk6si7742912wib.44 - gsmtp)
Oct 26 16:38:33 xx postfix/qmgr[477]: 2012B580A53: removed
Found the solution, apparently receive_override_options = no_address_mappings disables virtual aliases, so when I removed the line everything started to work as expected.