I try to configure a dialog with a Multivalue field containing a composite field.
However, I get the following error:
Invocation of method buttonClick in info.magnolia.ui.form.field.MultiField$1 failed.
As pointed out in a similar question on Magnolia Forum, the issue was in the configuration tree.
I had 2 configuration nodes at an incorrect level of the dialog's tree.
It's pretty deep so one must watch out for misconfiguration.
Solution:
Dialog config tree exported
<?xml version="1.0" encoding="UTF-8"?>
<sv:node sv:name="tabEvents" xmlns:sv="http://www.jcp.org/jcr/sv/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<sv:property sv:name="jcr:primaryType" sv:type="Name">
<sv:value>mgnl:contentNode</sv:value>
</sv:property>
<sv:property sv:name="jcr:uuid" sv:type="String">
<sv:value>90ad1aeb-ea0c-448b-b323-c727d6a47fc9</sv:value>
</sv:property>
<sv:property sv:name="jcr:createdBy" sv:type="String">
<sv:value>admin</sv:value>
</sv:property>
<sv:property sv:name="label" sv:type="String">
<sv:value>Events</sv:value>
</sv:property>
<sv:property sv:name="mgnl:activationStatus" sv:type="Boolean">
<sv:value>false</sv:value>
</sv:property>
<sv:property sv:name="mgnl:created" sv:type="Date">
<sv:value>2016-10-18T16:15:34.890+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:createdBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivated" sv:type="Date">
<sv:value>2016-10-18T16:15:34.891+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivatedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModified" sv:type="Date">
<sv:value>2016-10-18T16:16:13.118+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModifiedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:node sv:name="fields">
<sv:property sv:name="jcr:primaryType" sv:type="Name">
<sv:value>mgnl:contentNode</sv:value>
</sv:property>
<sv:property sv:name="jcr:uuid" sv:type="String">
<sv:value>c28bd951-0455-4fa8-a373-0d837c21e6c7</sv:value>
</sv:property>
<sv:property sv:name="jcr:createdBy" sv:type="String">
<sv:value>admin</sv:value>
</sv:property>
<sv:property sv:name="mgnl:activationStatus" sv:type="Boolean">
<sv:value>false</sv:value>
</sv:property>
<sv:property sv:name="mgnl:created" sv:type="Date">
<sv:value>2016-10-18T16:15:34.890+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:createdBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivated" sv:type="Date">
<sv:value>2016-10-18T16:15:34.892+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivatedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModified" sv:type="Date">
<sv:value>2016-10-18T16:15:43.016+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModifiedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:node sv:name="events">
<sv:property sv:name="jcr:primaryType" sv:type="Name">
<sv:value>mgnl:contentNode</sv:value>
</sv:property>
<sv:property sv:name="jcr:uuid" sv:type="String">
<sv:value>e778e569-0def-4416-8db3-c827c01229bf</sv:value>
</sv:property>
<sv:property sv:name="class" sv:type="String">
<sv:value>info.magnolia.ui.form.field.definition.MultiValueFieldDefinition</sv:value>
</sv:property>
<sv:property sv:name="jcr:createdBy" sv:type="String">
<sv:value>admin</sv:value>
</sv:property>
<sv:property sv:name="label" sv:type="String">
<sv:value>Events</sv:value>
</sv:property>
<sv:property sv:name="mgnl:activationStatus" sv:type="Boolean">
<sv:value>false</sv:value>
</sv:property>
<sv:property sv:name="mgnl:created" sv:type="Date">
<sv:value>2016-10-18T16:15:34.890+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:createdBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivated" sv:type="Date">
<sv:value>2016-10-18T16:15:34.892+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivatedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModified" sv:type="Date">
<sv:value>2016-10-18T16:16:30.044+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModifiedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="transformerClass" sv:type="String">
<sv:value>info.magnolia.ui.form.field.transformer.multi.MultiValueSubChildrenNodePropertiesTransformer</sv:value>
</sv:property>
<sv:node sv:name="field">
<sv:property sv:name="jcr:primaryType" sv:type="Name">
<sv:value>mgnl:contentNode</sv:value>
</sv:property>
<sv:property sv:name="jcr:uuid" sv:type="String">
<sv:value>696d2cb1-58a4-4851-a167-253b8a47ca72</sv:value>
</sv:property>
<sv:property sv:name="class" sv:type="String">
<sv:value>info.magnolia.ui.form.field.definition.CompositeFieldDefinition</sv:value>
</sv:property>
<sv:property sv:name="jcr:createdBy" sv:type="String">
<sv:value>admin</sv:value>
</sv:property>
<sv:property sv:name="mgnl:activationStatus" sv:type="Boolean">
<sv:value>false</sv:value>
</sv:property>
<sv:property sv:name="mgnl:created" sv:type="Date">
<sv:value>2016-10-18T16:15:34.890+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:createdBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivated" sv:type="Date">
<sv:value>2016-10-18T16:15:34.892+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivatedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModified" sv:type="Date">
<sv:value>2016-10-18T16:15:56.610+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModifiedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="transformerClass" sv:type="String">
<sv:value>info.magnolia.ui.form.field.transformer.composite.NoOpCompositeTransformer</sv:value>
</sv:property>
<sv:node sv:name="fields">
<sv:property sv:name="jcr:primaryType" sv:type="Name">
<sv:value>mgnl:contentNode</sv:value>
</sv:property>
<sv:property sv:name="jcr:uuid" sv:type="String">
<sv:value>283d22e5-add7-4183-a11d-d93976cd15e0</sv:value>
</sv:property>
<sv:property sv:name="jcr:createdBy" sv:type="String">
<sv:value>admin</sv:value>
</sv:property>
<sv:property sv:name="mgnl:activationStatus" sv:type="Boolean">
<sv:value>false</sv:value>
</sv:property>
<sv:property sv:name="mgnl:created" sv:type="Date">
<sv:value>2016-10-18T16:15:34.890+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:createdBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivated" sv:type="Date">
<sv:value>2016-10-18T16:15:34.892+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivatedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModified" sv:type="Date">
<sv:value>2016-10-18T16:15:56.610+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModifiedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:node sv:name="eventName">
<sv:property sv:name="jcr:primaryType" sv:type="Name">
<sv:value>mgnl:contentNode</sv:value>
</sv:property>
<sv:property sv:name="jcr:uuid" sv:type="String">
<sv:value>122bcb52-6d85-4fcc-b8e3-ea7ab531f8f5</sv:value>
</sv:property>
<sv:property sv:name="class" sv:type="String">
<sv:value>info.magnolia.ui.form.field.definition.TextFieldDefinition</sv:value>
</sv:property>
<sv:property sv:name="jcr:createdBy" sv:type="String">
<sv:value>admin</sv:value>
</sv:property>
<sv:property sv:name="label" sv:type="String">
<sv:value>Name</sv:value>
</sv:property>
<sv:property sv:name="mgnl:activationStatus" sv:type="Boolean">
<sv:value>false</sv:value>
</sv:property>
<sv:property sv:name="mgnl:created" sv:type="Date">
<sv:value>2016-10-18T16:15:34.890+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:createdBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivated" sv:type="Date">
<sv:value>2016-10-18T16:15:34.894+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivatedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModified" sv:type="Date">
<sv:value>2016-10-18T16:17:32.365+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModifiedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
</sv:node>
<sv:node sv:name="eventArtists">
<sv:property sv:name="jcr:primaryType" sv:type="Name">
<sv:value>mgnl:contentNode</sv:value>
</sv:property>
<sv:property sv:name="jcr:uuid" sv:type="String">
<sv:value>ca110ecb-1f72-4c8f-8d22-e83046bd9d14</sv:value>
</sv:property>
<sv:property sv:name="class" sv:type="String">
<sv:value>info.magnolia.ui.form.field.definition.TextFieldDefinition</sv:value>
</sv:property>
<sv:property sv:name="jcr:createdBy" sv:type="String">
<sv:value>admin</sv:value>
</sv:property>
<sv:property sv:name="label" sv:type="String">
<sv:value>Artists (i.e. Prince, The Fugees)</sv:value>
</sv:property>
<sv:property sv:name="mgnl:activationStatus" sv:type="Boolean">
<sv:value>false</sv:value>
</sv:property>
<sv:property sv:name="mgnl:created" sv:type="Date">
<sv:value>2016-10-18T16:15:34.890+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:createdBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivated" sv:type="Date">
<sv:value>2016-10-18T16:15:34.896+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastActivatedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModified" sv:type="Date">
<sv:value>2016-10-18T16:18:21.871+02:00</sv:value>
</sv:property>
<sv:property sv:name="mgnl:lastModifiedBy" sv:type="String">
<sv:value>superuser</sv:value>
</sv:property>
</sv:node>
</sv:node>
</sv:node>
</sv:node>
</sv:node>
</sv:node>
Template (FTL)
[#if content.events?has_content]
[#list cmsfn.children(content.events) as eventItem]
${eventItem.eventName!}: ${eventItem.eventArtists!}.<br/>
[/#list]
[/#if]
Resources
wiki.magnolia-cms.com/display/DEV/Multi+Value+Field
documentation.magnolia-cms.com/display/DOCS/Multivalue+field
similar question on Magnolia Forum
similar issue on Magnolia's ticketing system
Related
For example, ἐν or Ἐν are the same, but should be distinguished from ἕν/Ἓν. I've tried utf8_bin which seems to be the closest, but is also case sensitive.
mysql> select 'ἐν' = 'Ἐν' collate utf8mb4_0900_as_ci;
+----------------------------------------------+
| 'ἐν' = 'Ἐν' collate utf8mb4_0900_as_ci |
+----------------------------------------------+
| 1 |
+----------------------------------------------+
mysql> select 'ἐν' = 'ἕν' collate utf8mb4_0900_as_ci;
+----------------------------------------------+
| 'ἐν' = 'ἕν' collate utf8mb4_0900_as_ci |
+----------------------------------------------+
| 0 |
+----------------------------------------------+
I make docker image with mysql.
I created init.sql file to create table when starting docker file.
but after building docker image, table is not created.
I want to create table via DockerFile or docker-compose.
Here is source code in github.
https://github.com/jpskgc/article
I created int.sql file and docker-compose.yml to use this init.sqlfile.
article
├ db
│ └ init.sql
│
└ docker-compose.yml
//docker-compose.yml
version: '3'
services:
db:
image: mysql
ports:
- '3306:3306'
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: article
MYSQL_USER: docker
MYSQL_PASSWORD: docker
volumes:
- ./db:/docker-entrypoint-initdb.d
//init.sql
CREATE DATABASE IF NOT EXISTS article;
use article;
CREATE TABLE IF NOT EXISTS `articles` (`id` int NOT NULL AUTO_INCREMENT PRIMARY KEY, uuid varchar(36), `title` VARCHAR(100) NOT NULL,`content` TEXT NOT NULL) ENGINE=InnoDB DEFAULT CHARSET=utf8;
create table IF NOT EXISTS images (id int AUTO_INCREMENT NOT NULL PRIMARY KEY, article_uuid varchar(36), image_name varchar(50));
I want to create table in the process of docker-compose up --build.
After command docker-compose up --build and assess to 'http://localhost:3000/', there is error in backend, which suggesting that table does not exist.
api_1 | 2019/07/31 06:29:58 [Recovery] 2019/07/31 - 06:29:58 panic recovered:
api_1 | GET /api/articles HTTP/1.1
api_1 | Host: localhost:2345
api_1 | Accept: application/json, text/plain, */*
api_1 | Accept-Encoding: gzip, deflate, br
api_1 | Accept-Language: en-US,en;q=0.9
api_1 | Connection: keep-alive
api_1 | Origin: http://localhost:3000
api_1 | Referer: http://localhost:3000/
api_1 | User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
api_1 |
api_1 |
api_1 | Error 1146: Table 'article.articles' doesn't exist
api_1 | /app/main.go:90 (0xb1c6d1)
api_1 | main.func2: panic(err.Error())
api_1 | /go/src/github.com/gin-gonic/gin/context.go:147 (0x8f97c9)
api_1 | (*Context).Next: c.handlers[c.index](c)
api_1 | /go/src/github.com/gin-gonic/gin/recovery.go:83 (0x90d259)
api_1 | RecoveryWithWriter.func1: c.Next()
api_1 | /go/src/github.com/gin-gonic/gin/context.go:147 (0x8f97c9)
api_1 | (*Context).Next: c.handlers[c.index](c)
api_1 | /go/src/github.com/gin-gonic/gin/logger.go:240 (0x90c300)
api_1 | LoggerWithConfig.func1: c.Next()
api_1 | /go/src/github.com/gin-gonic/gin/context.go:147 (0x8f97c9)
api_1 | (*Context).Next: c.handlers[c.index](c)
api_1 | /go/src/github.com/gin-gonic/gin/gin.go:391 (0x9036c9)
api_1 | (*Engine).handleHTTPRequest: c.Next()
api_1 | /go/src/github.com/gin-gonic/gin/gin.go:352 (0x902dbd)
api_1 | (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
api_1 | /usr/local/go/src/net/http/server.go:2774 (0x6dcc77)
api_1 | serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
api_1 | /usr/local/go/src/net/http/server.go:1878 (0x6d8860)
api_1 | (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
api_1 | /usr/local/go/src/runtime/asm_amd64.s:1337 (0x45a090)
api_1 | goexit: BYTE $0x90 // NOP
api_1 |
api_1 | [GIN] 2019/07/31 - 06:29:58 | 500 | 123.5997ms | 172.18.0.1 | GET /api/articles
As alternative solution, I changed server side codes to create table.
_, err = db.Exec("CREATE DATABASE IF NOT EXISTS article;")
if err != nil {
panic(err)
}
_, err = db.Exec("use article;")
if err != nil {
panic(err)
}
_, err = db.Exec("CREATE TABLE IF NOT EXISTS `articles` (`id` int NOT NULL AUTO_INCREMENT PRIMARY KEY,uuid varchar(36), `title` VARCHAR(100) NOT NULL,`content` TEXT NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8;")
if err != nil {
panic(err)
}
_, err = db.Exec("create table IF NOT EXISTS images (id int AUTO_INCREMENT NOT NULL PRIMARY KEY, article_uuid varchar(36), image_name varchar(50)); ")
if err != nil {
panic(err)
}
I have a table with unique index on (col1, col2, col3)
I have a query on this table with
...
left join my_table t on t.col1=x.col1 and t.col2=x.col2 and t.col3 in (1,2,3)
When the list for col3 has only one item, the explain shows a eq_ref join
+----+-------------+----------+------------+--------+---------------+-----+---------+---------------------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+--------+---------------+-----+---------+---------------------+------+----------+-------+
| 1 | PRIMARY | my_table | NULL | eq_ref | veh | veh | 8 | x.col1,x.col2,const | 1 | 100.00 | NULL |
+----+-------------+----------+------------+--------+---------------+-----+---------+---------------------+------+----------+-------+
But with two or more items, the index is not used anymore
+----+-------------+----------+------------+------+---------------+------+---------+------+----------+----------+------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+------+---------------+------+---------+------+----------+----------+------------------------------------------------+
| 1 | PRIMARY | my_table | NULL | ALL | veh | NULL | NULL | NULL | 13705314 | 100.00 | Range checked for each record (index map: 0x3) |
+----+-------------+----------+------------+------+---------------+------+---------+------+----------+----------+------------------------------------------------+
An additional non unique index on col3 doesn't help
How can I set the index(es) that can be used by such a join condition ?
Or how can I rewrite the query ?
Edit: additionnal info
explain format=json
select x.*,ifnull(group_concat(t.col4 separator ','),'') col4
from
(select col1,col2,total from tmp_data limit 10) x
left join my_table t on t.col1=x.col1 and t.col2=x.col2 and t.col3 in (38,85)
group by x.col1,x.col2
order by x.total desc
{
"query_block": {
"select_id": 1,
"cost_info": {
"query_cost": "411271.71"
},
"ordering_operation": {
"using_temporary_table": true,
"using_filesort": true,
"grouping_operation": {
"using_filesort": true,
"nested_loop": [
{
"table": {
"table_name": "x",
"access_type": "ALL",
"rows_examined_per_scan": 10,
"rows_produced_per_join": 10,
"filtered": "100.00",
"cost_info": {
"read_cost": "10.50",
"eval_cost": "2.00",
"prefix_cost": "12.50",
"data_read_per_join": "240"
},
"used_columns": [
"col1",
"col2",
"total"
],
"materialized_from_subquery": {
"using_temporary_table": true,
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 2,
"cost_info": {
"query_cost": "12.50"
},
"table": {
"table_name": "tmp_data",
"access_type": "ALL",
"rows_examined_per_scan": 10,
"rows_produced_per_join": 10,
"filtered": "100.00",
"cost_info": {
"read_cost": "10.50",
"eval_cost": "2.00",
"prefix_cost": "12.50",
"data_read_per_join": "5K"
},
"used_columns": [
"col1",
"col2",
"total"
]
}
}
}
}
},
{
"table": {
"table_name": "t",
"access_type": "ALL",
"possible_keys": [
"veh",
"col3"
],
"rows_examined_per_scan": 13705314,
"rows_produced_per_join": 320380,
"filtered": "100.00",
"range_checked_for_each_record": "index map: 0x3",
"cost_info": {
"read_cost": "347183.21",
"eval_cost": "64076.00",
"prefix_cost": "411271.71",
"data_read_per_join": "7M"
},
"used_columns": [
"col1",
"col2",
"col3",
"col4"
]
}
}
]
}
}
}
}
create table tmp_data
(col1 mediumint(1) unsigned not null default 0,
col2 mediumint(1) unsigned not null,
total decimal(11,0) not null default 0,
key veh (col1,col2) using hash
) engine=memory default charset=latin1 collate=latin1_general_ci
create table my_table
(col1 mediumint(1) unsigned not null,
col2 mediumint(1) unsigned not null,
col3 smallint(1) unsigned not null,
col4 text collate latin1_general_ci,
unique key veh (col1,col2,col3),
key col3 (col3)
) engine=myisam default charset=latin1 collate=latin1_general_ci
Use InnoDB, not MyISAM or MEMORY. Most Optimizer improvements have gone only into InnoDB. (Caveat: I do not have proof that this will help.)
While making that change, change the unique key to PRIMARY KEY (col1,col2,col3). The "clustered" nature of InnoDB's PK should help further. (This change will have no impact in MyISAM.)
For a mysql database, I have some XML dump files.
One table import file for a pricing table looks like this:
<database>
<table>
<row>
<id>5954017</id>
<foo>narf</foo>
<bar_id>1377</bar_id>
<price_single>800.00</price_single>
<price_double>1500.00</price_double>
<price_triple>2000.00</price_triple>
<price_quad>1900.00</price_quad>
<currency>USD</currency>
</row>
...
</table>
</database>
It has
$ xmllint --xpath "count(//row)" import.xml
223198
rows and its size is:
du -h import.xml
69M import.xml
I want to import via mysql's LOAD XML feature. The table will be always be truncated beforehand.
MySQL [my_database]> LOAD XML LOCAL INFILE 'import.xml' INTO TABLE pricing ROWS IDENTIFIED BY '<row>' \G
It succeeds, yet that table import takes seemingly a rather long time:
Query OK, 223198 rows affected (1 hour 44 min 48.40 sec)
Records: 223198 Deleted: 0 Skipped: 0 Warnings: 0
I was reading that people are importing gigabytes of data with this LOAD INFILE feature, and I expected its performance to be much faster, in the realms of minutes, not hours. Is my expectation wrong? Is this a normal time for such a dataset of 200,000 entries?
(I also am comparing this speed to a custom php import script that parses the XML manually and inserts each row one by one; and that tasks takes 45 minutes for all the tables. I expected the LOAD XML LOCAL INFILE to outperform that task.)
My table looks like this:
MySQL [my_database]> DESCRIBE pricing;
+----------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| foo | varchar(256) | YES | | NULL | |
| bar_id | int(11) | YES | | NULL | |
| price_single | float | YES | | NULL | |
| price_double | float | YES | | NULL | |
| price_triple | float | YES | | NULL | |
| price_quad | float | YES | | NULL | |
| currency | varchar(3) | YES | | NULL | |
+----------------+--------------+------+-----+---------+-------+
And was created via:
DROP TABLE IF EXISTS `pricing`;
CREATE TABLE `pricing` (
`id` int(11) NOT NULL,
`foo` varchar(256) DEFAULT NULL,
`bar_id` int(11) DEFAULT NULL,
`price_single` float DEFAULT NULL,
`price_double` float DEFAULT NULL,
`price_triple` float DEFAULT NULL,
`price_quad` float DEFAULT NULL,
`currency` varchar(3) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
What can I do to improve the performance of LOAD XML LOCAL INFILE?
Transform your XML into a CSV file making the import almost instant.
You can transform your XML via xslt using e.g. xsltproc:
$ xsltproc transformToCsv.xsl price.xml > price.csv
"5954017"╡"narf"╡"1377"╡"800.00"╡"1500.00"╡"2000.00"╡"1900.00"╡"USD"
The appropriate xslt transformToCsv.xsl looks like this:
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text"
encoding="utf-8"/>
<xsl:param name="delim"
select="'╡'"/>
<xsl:param name="quote"
select="'"'"/>
<xsl:param name="break"
select="'
'"/>
<xsl:template match="/">
<xsl:apply-templates select="database/table/row"/>
</xsl:template>
<xsl:template match="row">
<xsl:apply-templates/>
<xsl:if test="following-sibling::*">
<xsl:value-of select="$break"/>
</xsl:if>
</xsl:template>
<xsl:template match="*">
<xsl:value-of select="concat($quote, normalize-space(), $quote)"/>
<xsl:if test="following-sibling::*">
<xsl:value-of select="$delim"/>
</xsl:if>
</xsl:template>
<xsl:template match="text()"/>
</xsl:stylesheet>
The import syntax then changes to:
$ mysql \
-h YOUR_MYSQL_HOST \
-P YOUR_PORT \
-uYOUR_USER \
-pYOUR_PASSWORD \
YOUR_DATABASE \
-e "LOAD DATA LOCAL INFILE 'pricing.csv' INTO TABLE pricing FIELDS TERMINATED BY '╡' ENCLOSED BY '\"' \G"
Importing 200,000 entries becomes a matter of (milli) seconds.
I installed hive-0.7.1 using MySQL 5.5 as the metastore. When I tried to create a simple table, it failed. I got exception
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (`hiveone`.`tbls`, CONSTRAINT `TBLS_FK2` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`))
Please help to advice how I should proceed.
hive> CREATE TABLE pokes (foo INT, bar STRING);
FAILED: Error in metadata: javax.jdo.JDODataStoreException: Insert of object "org.apache.hadoop.hive.metastore.model.MTable#4d092447" using statement "INSERT INTO `TBLS` (`TBL_ID`,`LAST_ACCESS_TIME`,`TBL_TYPE`,`CREATE_TIME`,`OWNER`,`SD_ID`,`RETENTION`,`VIEW_ORIGINAL_TEXT`,`TBL_NAME`,`DB_ID`,`VIEW_EXPANDED_TEXT`) VALUES (?,?,?,?,?,?,?,?,?,?,?)" failed : Cannot add or update a child row: a foreign key constraint fails (`hiveone`.`tbls`, CONSTRAINT `TBLS_FK2` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`))
NestedThrowables:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (`hiveone`.`tbls`, CONSTRAINT `TBLS_FK2` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`))
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive-site.xml settings as below:
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hiveone?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hadoop</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>pwd</value>
</property>
</configuration>
Tables in MySQL are listed below:
SDS table is empty.
mysql> show tables;
+-------------------+
| Tables_in_hiveone |
+-------------------+
| BUCKETING_COLS |
| COLUMNS |
| DATABASE_PARAMS |
| PARTITION_KEYS |
| SDS |
| SD_PARAMS |
| SEQUENCE_TABLE |
| SERDES |
| SERDE_PARAMS |
| SORT_COLS |
| TABLE_PARAMS |
| TBLS |
| dbs |
+-------------------+
13 rows in set (0.00 sec)
The question is what SDS data does Hive expect?
hive-0.7.1 doesn't support MySQL 5.5. End of the story.