I'm using NODEJS and MySQL for an API server and I ran into an issue. The queries that I'm running have multiple rows of results due to foreign keys.
User Table Address Table
|id| name| |id|user_id| Address|
| 1| frank| | 1| 1| 123 main street, Moncton|
| 2| steve| | 2| 2|97 duplessie street, Edmundston|
| 3| 2| 933 manhole av, Grand Sault|
How would I go about to get a multilevel json object like below for my query result, or at least, for my API response.
{
"id":1,
"name": "frank",
"addresses": [
{"id":1,"Address":"123 main street, Moncton"}
]
},
{
"id":2,
"name": "steve",
"addresses": [
{"id":2,"Address":"97 duplessie street, Edmundston"},
{"id":3,"Address":"933 manhole av, Grand Sault"}
]
}
I followed #dmfay direction and came up with a solution that works for me.
This a function where I pass the 2 JSON objects to merge addresses into users JSON and returns everything in a single JSON object.
function combineResults(users, addresses){
***********************************
*** Go through each user object ***
***********************************
users.forEach(function(user) {
user.addresses = addresses.find(address => address.user_id === user.id);
}
return users;
}
If you're just running raw queries, the results you get are the results you get. Object-relational mapping libraries like Objection, Bookshelf, or Sequelize will handle this for you automatically but may involve more setup and/or abstraction than you'd like.
If you'd rather keep it minimal, you can always process the result array yourself. I ran into this exact issue some time ago with my own project (a data mapper for Postgres) and developed a decomposer you should be able to cannibalize without issues. See the docs for constructing schemas.
Related
this is the code to insert json in my sql
function insert_ema($json){
$a= new Sql();
$b=$a->connection;
$sql = "INSERT INTO ema (ema_ten) VALUES ('$json')";
if ($b->query($sql) === TRUE) {
echo PHP_EOL." New record created successfully \n";
} else {
echo PHP_EOL." Error: " . $sql . "<br>" . $b->error;
}
$b->close();
;}
insert_ema('{"firstName":"John", "lastName":"Doe","3":"Jo", "4":"Do"}');
+----------------------------------------------------------------+----+
| ema_ten | id |
+----------------------------------------------------------------+----+
| {"3": "Jo", "4": "Do", "lastName": "Doe", "firstName": "John"} | 1 |
| {"3": "Jo", "4": "Do", "lastName": "Doe", "firstName": "John"} | 2 |
+----------------------------------------------------------------+----+
the sql saved above is in reverse order!! how can i fix it
The reason why I want to persevere order is, I want to be able to convert the json to an array and use pop .
I think MySQL should save arrays and also sort this issue.
https://dev.mysql.com/doc/refman/8.0/en/json.html says:
To make lookups more efficient, MySQL also sorts the keys of a JSON object. You should be aware that the result of this ordering is subject to change and not guaranteed to be consistent across releases.
This mean you should not depend on any particular sort order of the keys in a JSON object. JSON arrays have order, but the keys of JSON objects don't.
JSON objects are equal if their keys and respective values are the same, regardless of order:
mysql> select cast('{"firstName":"John", "lastName":"Doe","3":"Jo", "4":"Do"}' as json)
= cast('{"3": "Jo", "4": "Do", "lastName": "Doe", "firstName": "John"}' as json)
as is_equal;
+----------+
| is_equal |
+----------+
| 1 |
+----------+
Re your comment:
The point of the above example is that you can't make MySQL store keys in your intended order. MySQL's implementation of JSON doesn't do that. It rearranges JSON object keys to make it more efficient for lookups. You don't get a say in this.
JSON arrays can be ordered. So your only option to preserve order is to use an array, where each element of the array is an object with a single key:
[{"firstName":"John"}, {"lastName":"Doe"}, {"3":"Jo"}, {"4":"Do"}]
I understand this is not what you asked for, but what you asked for cannot be achieved in MySQL.
I create a job in Talend to extract data from an API to MySQL table.
I used tRest as input(data pull), and tLogRow for now to view it in table form.
I lost it at mapping the json response into column for each of the field and value.
but the response was in one string line, when it's prettier formatted:
{
"data":
{
"user":
[
{"id":8, "name":"Alex"},
{"id":9, "name":"John"}
]
},
"http_code":200,
"message":[]
}
how can I turn this response and map into table like below
id
name
8
Alex
9
John
Glad if you guys have the working example too. Or using different components might help?
Here's how to set up tExtractJSONFields:
Loop path query needs to be:
$.data.user[*]
And just use the attribute names as json query for for each column.
The output:
.--+----.
|tLogRow_1|
|=-+---=|
|id|name|
|=-+---=|
|8 |Alex|
|9 |John|
'--+----'
The data in the column looks like the below:
{
"activity": {
"token": "e7b64be4-74d4-7a6d-a74b-xxxxxxx",
"route": "http://example.com/enroll/confirmation",
"url_parameters": {
"Success": "True",
"ContractNumber": "003992314W",
"Barcode": "1908Y10Z",
"price": "8.99"
},
"server_info": {
"cookie": [
"_ga=xxxx; _fbp=xxx; _hjid=xxx; XDEBUG_SESSION=XDEBUG_ECLIPSE;"
],
"upgrade-insecure-requests": [
"1"
],
},
"campaign": "Unknown/None",
"ip": "192.168.10.1",
"entity": "App\\Models\\User",
"entity_id": "1d9f3066-13ce-4659-b10d-xxxxx",
},
"time": "2021-05-21 20:15:02"
}
My code that I am using is below:
SELECT *
FROM websote.stored_events
WHERE JSON_EXTRACT(event_properties, '$.route') = 'http://example.com/enroll/confirmation'
ORDER BY created_at DESC LIMIT 500;
The code works on the other the json values just not the url ones. I've tried escaping the values in MySQL like the below:
SELECT *
FROM websote.stored_events
WHERE JSON_EXTRACT(event_properties, '$.route') = 'http:///example.com//enroll//confirmation'
ORDER BY created_at DESC LIMIT 500;
But still no luck. Any help on this would be appreciated!
Route is a nested property; I would have expected the path to be
JSON_EXTRACT(event_properties, '$.activity.route')
Your example data isn't valid JSON. You can't have a comma after the last element in an object or array:
"entity_id": "1d9f3066-13ce-4659-b10d-xxxxx",
},
^ here
If I remove that and other similar cases, I can test your data inserts into a JSON column and I can extract the object element you described:
mysql> select json_extract(event_properties, '$.activity.route') as route from stored_events;
+------------------------------------------+
| route |
+------------------------------------------+
| "http://example.com/enroll/confirmation" |
+------------------------------------------+
Note the value is returned with double-quotes. This is because it's returned as a JSON document, a scalar string. If you want the raw value, you have to unquote it:
mysql> select json_unquote(json_extract(event_properties, '$.activity.route')) as route from stored_events;
+----------------------------------------+
| route |
+----------------------------------------+
| http://example.com/enroll/confirmation |
+----------------------------------------+
If you want to search for that value, you would have to do a similar expression:
select * from stored_events
where json_unquote(json_extract(event_properties, '$.activity.route'))
= 'http://example.com/enroll/confirmation'
Searching based on object properties stored in JSON has disadvantages.
It requires complex expressions that force you (and anyone else you needs to maintain your code) to learn a lot of details about how JSON works.
It cannot be optimized with an index. This query will run a table-scan. You can add virtual columns with indexes, but that adds to complexity and if you need to ALTER TABLE to add virtual columns, it misses the point of JSON to store semi-structured data.
The bottom line is that if you find yourself using JSON functions in the WHERE clause of a query, it's a sign that you should be storing the column you want to search as a normal column, not as part of a JSON document.
Then you can write code that is easy to read, easy for your colleagues to maintain, and can be optimized easily with indexes:
SELECT * FROM stored_events
WHERE route = 'http://example.com/enroll/confirmation';
You can still store other properties in the JSON document, but the ones you want to be searchable should be stored in normal columns.
You might like to view my presentation How to Use JSON in MySQL Wrong.
I need to download application JAR for execution, and I managed to figure it out up to this step part in my script:
jq '.files[] | select(.uri | contains("RELEASE") and contains(".jar"))'
This gives me a bunch of results that looks like multiple blocks of this:
{
other versions
}
{
"uri": "/2.0.6-RELEASE/app-name-2.0.6-RELEASE.jar",
"size": 32981192,
"lastModified": "2020-02-05T14:21:06.728-05:00",
"folder": false,
"sha1": "whatever",
"sha2": "whatever2",
"mdTimestamps": {
"properties": "2020-02-05T14:21:13.468-05:00"
}
}
Based on the documentation, I tried to extend this to:
jq '.files[] | select(.uri | contains("RELEASE") and contains(".jar")) | max_by(.lastModified)'
But it returns error
jq: error (at <stdin>:3284): Cannot index string with string "lastModified"
Documentation is vague on combining all the different search filters together.. I am not sure what is the correct syntax, please.
My goal: Retrieve only one element, ideally just the URI string so that I can use that to fetch the JAR directly on my next command without further processing.
I also tried the documented syntax for max_by with suffix: | {uri} - same error.
max_by requires an array as input, so in the spirit of your attempt, you might wish to consider:
.files
| map( select(.uri | contains("RELEASE")
and contains(".jar")))
| max_by(.lastModified)
p.s.
In future, please follow the guidelines at
http://stackoverflow.com/help/mcve
Let's take a simple schema with two tables, one that describes an simple entity item (id, name)
id | name
------------
1 | foo
2 | bar
and another, lets call it collection, that references to an item, but inside a JSON Object in something like
{
items: [
{
id: 1,
quantity: 2
}
]
}
I'm looking for a way to eventually enrich this field (kind of like populate in Mongo) in the collection with the item element referenced, to retrieve something like
{
...
items: [
{
item: {
id: 1,
name: foo
},
quantity: 2
}
]
...
}
If you have a solution with PostgreSQL, I take it as well.
If I understood correctly, your requirement is to convert an Input JSON data into MySQL table so that you can work with JSON but leverage the power of SQL.
Mysql8 recently released JSONTABLE function. By using this function, you can store your JSON in the table directly and then query it like any other SQL query.
It should serve your immediate case, but this means that your table schema will have a JSON column instead of traditional MySQL columns. You will need to check if it serves your purpose.
This is a good tutorial for the same.