I'm using cygnus to store data in both MySQL Database and Cosmos.
Storing the data in cosmos works ok, but when storing in MySQL, especifically with the attribute "cygnusagent.sinks.mysql-sink.attr_persistence = column" instead of "cygnusagent.sinks.mysql-sink.attr_persistence = row" the data is not stored and I'm getting some errors in flume log.
As the table needs to be created previously when using the column attribute(more info : https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/design/OrionMySQLSink.md#important-notes-regarding-the-persistence-mode), I create the table:
CREATE TABLE def_servpath_sensorreading4_sensorreading(systemid int,value float, sensorid int, nodeid int);
DESCRIBE def_servpath_sensorreading4_sensorreading(systemid int,value float, sensorid int, nodeid int);
mysql> DESCRIBE def_servpath_sensorreading4_sensorreading;
+----------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+---------+------+-----+---------+-------+
| systemid | int(11) | YES | | NULL | |
| value | float | YES | | NULL | |
| sensorid | int(11) | YES | | NULL | |
| nodeid | int(11) | YES | | NULL | |
+----------+---------+------+-----+---------+-------+
4 rows in set (0.00 sec)
This is the flume log:
11 Aug 2015 11:40:31,977 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.persist:240) - [mysql-sink] Persisting data at OrionMySQLSink. Database: def_serv, Table: def_servpath_sensorreading3_sensorreading, Timestamp: 2015-08-11T08:40:31.969, Data (attrs): {systemid=1, value=29.2, sensorid=2, nodeid=1}, (metadata): {sensorid_md=[], systemid_md=[], value_md=[], nodeid_md=[]}
11 Aug 2015 11:40:31,978 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:182) - Bad context data (Unknown column 'systemid' in 'field list')
11 Aug 2015 11:40:31,978 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1439278647-655-0000000005)
11 Aug 2015 11:40:32,369 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356) - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/sensorreading3_sensorreading/sensorreading3_sensorreading.txt), Data ({"recvTime":"2015-08-11T08:40:31.969Z","nodeid":"1", "nodeid_md":[],"sensorid":"2", "sensorid_md":[],"systemid":"1", "systemid_md":[],"value":"29.2", "value_md":[]})
11 Aug 2015 11:40:32,858 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1439278647-655-0000000005)
Where im getting the following error:
Bad context data (Unknown column 'systemid' in 'field list')
Do I need to create any more columns in the table related with the metadata? Am I creating the table correctly?
Thanks in advance.
Yes, this section of the documentation details that an additional column regarding the metadata must be added per each attribute (by sufixing the attribute's name with _md). Nevertheless, an example is not given (something to be improved, btw), so here it is (tailored to your specific case):
create table def_servpath_sensorreading4_sensorreading(recvTime text, systemid int, systemid_md text, value float, value_md text, sensorid int, sensorid_md text, nodeid int, nodeid_md text);
It is very important the attribute names match the fields in the table. I mean, your entity must have an attribute named systemid; if your attribute would be named system_id then that had to be the field name as well (being the metadata field system_id_md).
Ass you can see, there is an addition column regarding the reception time (recvTime), since Cygnus persists such a value each time a notification is received.
Regarding the table name, you have composed it perfectly: it must be the concatenation of the notified FIWARE service-path, the entity ID and the entity type ('_' as concatenation character). Such a table must exists within a databased named as the notified FIWARE service.
Related
I have a large amount of JSON data that needs to be inserted into a MySQLx Collection table. The current Node implementation keeps crashing when I attempt to load my JSON data in, and I suspect it's because I'm inserting too much at once through the collection API. I'd like to manually insert the data into the database using a traditional SQL statement (in the hope that they will get me pass this NodeJs crash).
The problem is that I have this table def:
+--------------+---------------+------+-----+---------+-------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------+------+-----+---------+-------------------+
| doc | json | YES | | NULL | |
| _id | varbinary(32) | NO | PRI | NULL | STORED GENERATED |
| _json_schema | json | YES | | NULL | VIRTUAL GENERATED |
+--------------+---------------+------+-----+---------+-------------------+
But when running
insert into documents values ('{}', DEFAULT, DEFAULT)
I get:
ERROR 3105 (HY000): The value specified for generated column '_id' in table 'documents' is not allowed.
I've tried with not providing the DEFAULTs, with NULL (but _id doesn't allow NULL even though that's the default), with 0 for _id, with numbers and with uuid_to_bin(uuid()) but I still get the same error.
How can I insert this data into the table directly (I'm using session.sql('INSERT...').bind(JSON.stringify(data)).execute() - using the #mysql/xdevapi library)
The _id column is auto generated from the value of the namesake field in the JSON document. When you use the CRUD interface to insert documents, the X Plugin is capable of generating a unique value for this field. However, by executing a plain SQL statement, you are also by-passing that logic. So, you are able to insert documents if you generate the _ids yourself, otherwise you will bump into that error.
As an example (using crypto.randomInt()):
const { randomInt } = require('crypto')
session.sql('insert into documents (doc) values (?)')
.bind(JSON.stringify({ _id: randomInt(Math.pow(2, 48) - 1) }))
.execute()
Though I'm curious about the issue with the CRUD API and I wanted to see if I was able to reproduce it as well. How are you inserting those documents in that case and what kind of feedback (if any) is provided when it "crashes"?
Disclaimer: I'm the lead developer of the MySQL X DevAPI connector for Node.js
I have received a Mysql table with customer data which is pretty badly structured, almost all the fields are TEXT. To prevent losing data I have created another table where I am trying to import the columns in a correct and useful manner. Splitting Full_name into name and surname columns work great but when I try to convert the field created_time containing dd/mm/yy date to DATETIME it shows wrong data eg. 10/10/18 to 2010-10-18.
I have resorted to creating another field created_time_old, copying the text data there and then converting it via STR_TO_DATE as some other answer suggested. However, I can only put strings there, when I put the whole column it just gives me:
1411 - Incorrect datetime value: '' for function str_to_date.
I assume that there is another way/function that will manage to do it but I am not very experienced when it comes to SQL. Moreover, if you have any other suggestions when it comes to my code, please post them :)
My code is below
TRUNCATE TABLE Facetel_bazaPGS;
INSERT INTO Facetel_bazaPGS (id,created_time_old,campaign_name,email,Imię,Nazwisko,`phone_number`,platform,Time_added)
SELECT `id`,created_time,`campaign_name`,`email`,substring_index(`full_name`, ' ',1 ),substring(`full_name` from INSTR(`full_name`, ' ') + 1),`phone_number`,`platform`,Time_added
FROM `Facetel_bazaPGS_input`;
UPDATE Facetel_bazaPGS
SET platform = Replace(REPLACE(platform, 'fb', 'Facebook') , 'ig', 'Instagram');
UPDATE Facetel_bazaPGS
SET created_time = STR_TO_DATE(`created_time_old`, '%d/%m/%y');
EDIT#1: Adding sample data (can't give real data because of GDPR)
+------------------+--------------+---------------+--------------------+--------------+---------------------+----------+---------------------+
| id | created_time | campaign_name | email | full_name | phone_number | platform | Time_added |
+------------------+--------------+---------------+--------------------+--------------+---------------------+----------+---------------------+
| 1010334092505681 | 10/10/18 | leady | samplemail#mail.eu | Name | your_typical_number | ig | 2018-10-11 08:29:45 |
| 1010457652493325 | 10/10/18 | leady | samplemail#mail.eu | Name Surname | your_typical_number | ig | 2018-10-11 08:29:45 |
| 1010470612492029 | 10/10/18 | leady | samplemail#mail.eu | Name Surname | your_typical_number | fb | 2018-10-11 08:29:45 |
+------------------+--------------+---------------+--------------------+--------------+---------------------+----------+---------------------+
This answer is a speculation, but it attempts to get to the root cause of your 1411 error coming from the call to STR_TO_DATE. We can try running the following query to detect malformed date strings in your Facetel_bazaPGS_input source table:
SELECT *
FROM Facetel_bazaPGS_input
WHERE created_time NOT REGEXP '^[0-9]{2}/[0-9]{2}/[0-9]{2}$';
This would return any record not having a created_time in the format you expect. If you don't see an empty result set, then you can fix the date strings.
Note: Just now I can see that your error message seems to be saying that some created_time values are either empty string or NULL. In either case, the above query should flush out such records.
Thank you again for all your suggestions, I have given up, changed the format of the column in Excel, imported again and asked for any future imputs to have YYYY-MM-DD format in the created_time column.
I have two columns in a mysql table that are set by user preference:
custom_string VARCHAR(255) NOT NULL
items_per_row TINYINT UNSIGNED NOT NULL DEFAULT '5'
EXAMPLE:
+--------------------------------------+
| preferences |
+--------------------------------------+
| id | custom_string | items_per_group |
+--------------------------------------+
| 1 | TRINKETS | 8 |
+--------------------------------------+
| 2 | | 5 |
+--------------------------------------+
| 3 | MYSTUFF | 7 |
+--------------------------------------+
items_per_row is a required field. The custom_string field is optionally used to personalize the way grouped items in the list are displayed.
The user is able to update these preferences any time. Here is a crude example of how items might be displayed:
I am wanting to find a way to constrain the length of each user's custom_string so that if it is not blank it must have a length that is exactly the same as the corresponding items_per_group value. I can easily validate the user input with both javascript and PHP and prevent the data from being entered into the database if it doesn't comply with this requirement, however, is there a way to set this constraint within mysql so that an attempt to have an 'invalid' string would be rejected?
MySQL does not implement check constraints. With them, this would be easy:
alter table preferences add constraint chk_preferences_custom
check (custom_string is null or length(custom_string) = items_per_group);
Your only option in MySQL is to use a trigger for this purpose.
In practice, it might be simpler to check at the application level when you insert/update custom_string.
I have a table that basically looks like the following:
Timestamp | Service | Observation
----------+---------+------------
... | vm-1 | 15
... | vm-1 | 20
... | vm-1 | 20
... | vm-1 | 20
... | vm-1 | 20
... | vm-1 | 20
... | bvm-2 | 184
... | bvm-2 | 104
... | bvm-2 | 4
... | bvm-2 | 14
... | bvm-2 | 657
... | bvm-2 | 6
... | bvm-2 | 6
The Service column will not have a lot of different values. I don't know at table creation time what all possible values are going to be so I can't use an enum, but the number of distinct values are going to grow very slowly at (less than ~10 new distinct values per month or less), whereas I'll have thousands of new observations per day.
Right now I'm just thinking of using a VARCHAR or mysql's TEXT type for the Service column, but given the specifics of the situation those kind of seem wasteful.
Are databases usually smart about this sort of thing? Or is there some way I can hint to the database that this behavior is something that it can reliably exploit?
I'm using MySQL 5.7. I'd prefer something standards compliant or portable, but I'm also open to MySQL specific workarounds.
EDIT:
In other words, what I want is for the column to be treated like an enum, but have the database figure out dynamically based on the data that shows up in the table what the different enum values are.
Every time you need to use an enum you should consider creating another table and reference to it. It's basic normalization. So create one table for the ServiceType with a name and an id field the name can be VARCHAR and the id should be INT. The actual table then just uses the id instead of the service name.
You can write a simple stored procedure to do the inserting and looking up of duplicate names as well as a view to access the results so outside of the DB you barely know how it is internally handled.
Your stored procedure needs to:
Check if the service exists and insert it if not. INSERT IGNORE ... is probably your friend here.
Get the ID of the service with SELECT id INTO #serv_id FROM ServiceType WHERE name = [service_name];
Insert into the table with the service ID instead of the service.
Don't over optimize. MySQL does not store TINYINT more efficiently than INT so just use the latter and it won't fail until you have billions of services.
I think , you have to create a new table for store the services and and then this table primary key (service_id) can be replaced in place of service text. But main table service column should be int type for storing the service id . So please change the service column type to int(4) .
hope it will be helpfull
I'm writing a program, in C++, to access tables in MySQL via the MySql C++ Connector.
I retrieve a record from the User (via GUI or Xml file).
Here are my questions:
Should I search the table first for
the given record, then append if it
doesn't exist,
Or append the record, and let MySQL
append the record if it is unique?
Here is my example table:
mysql> describe ing_titles;
+----------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+----------+------+-----+---------+-------+
| ID_Title | int(11) | NO | PRI | NULL | |
| Title | char(32) | NO | | NULL | |
+----------+----------+------+-----+---------+-------+
In judgment, I am looking for a solution that will enable my program to respond quickly to the User.
During development, I have small tables (less than 5 records), but I am expecting them to grow when I formally release the application.
FYI: I am using Visual Studion 2008, C++, wxWidgets, and MySQL C++ Connector on Windows XP and Vista.
Mark the field in question with a UNIQUE constraint and use INSERT ... ON DUPLICATE KEY UPDATE or INSERT IGNORE.
The former will update the records if they already exists, the latter will just do nothing.
Searching the table first is not efficient, since it requires two roundtrips to the server: the first one to search, the second one to insert (or to update).
The syntaxes above do the same in one sentence.