I currently have an application that uses a third party API whose endpoints return JSON. When a component on the front-end is mounted, I execute a function which makes a GET request to my own back-end and in return my back-end makes a GET request to the third party API, then the response from the API is returned as JSON to the front-end.
I have a limited amount of allowed requests to that API so I want to make sure to save the response to my database so that when a future requests are made, my back-end would return what's in my database instead of making a whole new GET request.
I'm not sure if storing JSON is wise or possible and this is why I decided to ask. Under what data type should the JSON be saved and would there be any drawbacks to what I'm doing?
Yes you can save JSON to mysql database. Mysql added the JSON datatype above 5.7 version.
Please refer http://www.mysqltutorial.org/mysql-json/
Yes, this is possible. MySQL implements JSON datatype since 5.7 version.
If You are asking about technical details about how to operate with this datatype, here is excellent shortcut.
Just quoting few examples:
CREATING:
mysql> CREATE TABLE facts (sentence JSON);
INSERTING:
mysql> INSERT INTO facts VALUES
> ('{"mascot": "Our mascot is a dolphin named \\"Sakila\\"."}');
READING:
mysql> SELECT sentence->"$.mascot" FROM facts;
But I bet that a real question is about how wise it is to store a JSON in database.
So the general answer is:
if developers of particular RDBMS included such aproach in their implementation, it is intended and desired for use.
So, as long as it is good idea to format your data as a JSON at all, it should be also a good idea to store this data in JSON column in RDBMS. I do not have an experience in that particular implementation (prefer Postgresql rather than MySQL), but I had started using JSON datatype as soon as I've needed it and still I do not consider it as a bad decision or something.
Epecially, when you consider storing JSON formated data inside a file and hooking just paths inside database, using a JSON type instead should be a good idea. Almost always storing JSON formated data in files WILL be slower than inserting and querying JSON, Especially, when You need access only to a particular key-value pairs (you can query just a particular keys in ordinary selects).
HOWEVER, when Your data is not inteded to be stored as a JSON format AT ALL, it will be a bad idea to use a JSON datatype also. What kind of data does JSON not like? Basically, all sorts of unstructured streams, when a number-of-keys :TO: overall-size ratio is very small. An example would be a dictionary with one key, and value storing 500 kByte long string:
{"file": "a very very very ... long string, perhaps just encoded file"}
In such case - yes, a better aproach is to store it as a regular files.
So as always, it all depends on a particular use case :)
Yes, you can save JSON response inside the table itself.
But It will be in simple string format and parsing of JSON string required to be done at code level.
You can use CLOB for the same.
It would just be a BLOB and theres nothing inheriently wrong with that, its a supported datatype.
However, i would urge you to save the JSON to a file and keep a list of paths instead. simply because it will be quicker.
I thought this was a n00b thing to do. And, so, I've never done it. Then I saw that FriendFeed did this and actually made their DB scale better and decreased latency. I'm curious if I should do this. And, if so, what's the right way to do it?
Basically, what's a good place to learn how to store everything in MySQL as a CouchDB sort of DB? Storing everything as JSON seems like it'd be easier and quicker (not to build, less latency).
Also, is it easy to edit, delete, etc., things stored as JSON on the DB?
Everybody commenting seems to be coming at this from the wrong angle, it is fine to store JSON code via PHP in a relational DB and it will in fact be faster to load and display complex data like this, however you will have design considerations such as searching, indexing etc.
The best way of doing this is to use hybrid data, for example if you need to search based upon datetime MySQL (performance tuned) is going to be a lot faster than PHP and for something like searching distance of venues MySQL should also be a lot faster (notice searching not accessing). Data you do not need to search on can then be stored in JSON, BLOB or any other format you really deem necessary.
Data you need to access is very easily stored as JSON for example a basic per-case invoice system. They do not benefit very much at all from RDBMS, and could be stored in JSON just by json_encoding($_POST['entires']) if you have the correct HTML form structure.
I am glad you are happy using MongoDB and I hope that it continues to serve you well, but don't think that MySQL is always going to be off your radar, as your app increases in complexity you may well end up needing an RDBMS for some functionality and features (even if it is just for retiring archived data or business reporting)
MySQL 5.7 Now supports a native JSON data type similar to MongoDB and other schemaless document data stores:
JSON support
Beginning with MySQL 5.7.8, MySQL supports a native JSON type. JSON values are not stored as strings, instead using an internal binary format that permits quick read access to document elements. JSON documents stored in JSON columns are automatically validated whenever they are inserted or updated, with an invalid document producing an error. JSON documents are normalized on creation, and can be compared using most comparison operators such as =, <, <=, >, >=, <>, !=, and <=>; for information about supported operators as well as precedence and other rules that MySQL follows when comparing JSON values, see Comparison and Ordering of JSON Values.
MySQL 5.7.8 also introduces a number of functions for working with JSON values. These functions include those listed here:
Functions that create JSON values: JSON_ARRAY(), JSON_MERGE(), and JSON_OBJECT(). See Section 12.16.2, “Functions That Create JSON Values”.
Functions that search JSON values: JSON_CONTAINS(), JSON_CONTAINS_PATH(), JSON_EXTRACT(), JSON_KEYS(), and JSON_SEARCH(). See Section 12.16.3, “Functions That Search JSON Values”.
Functions that modify JSON values: JSON_APPEND(), JSON_ARRAY_APPEND(), JSON_ARRAY_INSERT(), JSON_INSERT(), JSON_QUOTE(), JSON_REMOVE(), JSON_REPLACE(), JSON_SET(), and JSON_UNQUOTE(). See Section 12.16.4, “Functions That Modify JSON Values”.
Functions that provide information about JSON values: JSON_DEPTH(), JSON_LENGTH(), JSON_TYPE(), and JSON_VALID(). See Section 12.16.5, “Functions That Return JSON Value Attributes”.
In MySQL 5.7.9 and later, you can use column->path as shorthand for JSON_EXTRACT(column, path). This works as an alias for a column wherever a column identifier can occur in an SQL statement, including WHERE, ORDER BY, and GROUP BY clauses. This includes SELECT, UPDATE, DELETE, CREATE TABLE, and other SQL statements. The left hand side must be a JSON column identifier (and not an alias). The right hand side is a quoted JSON path expression which is evaluated against the JSON document returned as the column value.
See Section 12.16.3, “Functions That Search JSON Values”, for more information about -> and JSON_EXTRACT(). For information about JSON path support in MySQL 5.7, see Searching and Modifying JSON Values. See also Secondary Indexes and Virtual Generated Columns.
More info:
https://dev.mysql.com/doc/refman/5.7/en/json.html
CouchDB and MySQL are two very different beasts. JSON is the native way to store stuff in CouchDB. In MySQL, the best you could do is store JSON data as text in a single field. This would entirely defeat the purpose of storing it in an RDBMS and would greatly complicate every database transaction.
Don't.
Having said that, FriendFeed seemed to use an extremely custom schema on top of MySQL. It really depends on what exactly you want to store, there's hardly one definite answer on how to abuse a database system so it makes sense for you. Given that the article is very old and their main reason against Mongo and Couch was immaturity, I'd re-evaluate these two if MySQL doesn't cut it for you. They should have grown a lot by now.
json characters are nothing special when it comes down to storage, chars such as
{,},[,],',a-z,0-9.... are really nothing special and can be stored as text.
the first problem your going to have is this
{
profile_id: 22,
username: 'Robert',
password: 'skhgeeht893htgn34ythg9er'
}
that stored in a database is not that simple to update unless you had your own proceedure and developed a jsondecode for mysql
UPDATE users SET JSON(user_data,'username') = 'New User';
So as you cant do that you would Have to first SELECT the json, Decode it, change it, update it, so in theory you might as well spend more time constructing a suitable database structure!
I do use json to store data but only Meta Data, data that dont get updated often, not related to the user specific.. example if a user adds a post, and in that post he adds images ill parse the images and create thumbs and then use the thumb urls in a json format.
To illustrate how difficult it is to get JSON data using a query, I will share the query I made to handle this.
It doesn't take into account arrays or other objects, just basic datatypes. You should change the 4 instances of column to the column name storing the JSON, and change the 4 instances of myfield to the JSON field you want to access.
SELECT
SUBSTRING(
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', ''),
LOCATE(
CONCAT('myfield', ':'),
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', '')
) + CHAR_LENGTH(CONCAT('myfield', ':')),
LOCATE(
',',
SUBSTRING(
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', ''),
LOCATE(
CONCAT('myfield', ':'),
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', '')
) + CHAR_LENGTH(CONCAT('myfield', ':'))
)
) - 1
)
AS myfield
FROM mytable WHERE id = '3435'
This is an old question, but I am still able to see this at the top of the search result of Google, so I guess it would be meaningful to add a new answer 4 years after the question is asked.
First of all, there is better support in storing JSON in RDBMS. You may consider switching to PostgreSQL (although MySQL has supported JSON since v5.7.7). PostgreSQL uses very similar SQL commands as MySQL except they support more functions. One of the functions they added is that they provide JSON data type and you are now able to query the JSON stored. (Some reference on this) If you are not making up the query directly in your program, for example, using PDO in php or eloquent in Laravel, all you need to do is just to install PostgreSQL on your server and change database connection settings. You don't even need to change your code.
Most of the time, as the other answers suggested, storing data as JSON directly in RDBMS is not a good idea. There are some exception though. One situation I can think of is a field with variable number of linked entry.
For example, for storing tag of a blog post, normally you will need to have a table for blog post, a table of tag and a matching table. So, when the user wants to edit a post and you need to display which tag is related to that post, you will need to query 3 tables. This will damage the performance a lot if your matching table / tag table is long.
By storing the tags as JSON in the blog post table, the same action only requires a single table search. The user will then be able to see the blog post to be edit quicker, but this will damage the performance if you want to make a report on what post is linked to a tag, or maybe search by tag.
You may also try to de-normalize the database. By duplicating the data and storing the data in both ways, you can receive benefit of both method. You will just need a little bit more time to store your data and more storage space (which is cheap comparing to the cost of more computing power)
It really depends on your use case. If you are storing information that has absolutely no value in reporting, and won't be queried via JOINs with other tables, it may make sense for you to store your data in a single text field, encoded as JSON.
This could greatly simplify your data model. However, as mentioned by RobertPitt, don't expect to be able to combine this data with other data that has been normalized.
I would say the only two reasons to consider this are:
performance just isn't good enough with a normalised approach
you cannot readily model your particularly fluid/flexible/changing data
I wrote a bit about my own approach here:
What scalability problems have you encountered using a NoSQL data store?
(see the top answer)
Even JSON wasn't quite fast enough so we used a custom-text-format approach. Worked / continues to work well for us.
Is there a reason you're not using something like MongoDB? (could be MySQL is "required"; just curious)
Here is a function that would save/update keys of a JSON array in a column and another function that retrieves JSON values. This functions are created assuming that the column name of storing the JSON array is json. It is using PDO.
Save/Update Function
function save($uid, $key, $val){
global $dbh; // The PDO object
$sql = $dbh->prepare("SELECT `json` FROM users WHERE `id`=?");
$sql->execute(array($uid));
$data = $sql->fetch();
$arr = json_decode($data['json'],true);
$arr[$key] = $val; // Update the value
$sql=$dbh->prepare("UPDATE `users` SET `json`=? WHERE `id`=?");
$sql->execute(array(
json_encode($arr),
$uid
));
}
where $uid is the user's id, $key - the JSON key to update and it's value is mentioned as $val.
Get Value Function
function get($uid, $key){
global $dbh;
$sql = $dbh->prepare("SELECT `json` FROM `users` WHERE `id`=?");
$sql->execute(array($uid));
$data = $sql->fetch();
$arr = json_decode($data['json'], true);
return $arr[$key];
}
where $key is a key of JSON array from which we need the value.
It seems to me that everyone answering this question is kind-of missing the one critical issue, except #deceze -- use the right tool for the job. You can force a relational database to store almost any type of data and you can force Mongo to handle relational data, but at what cost? You end up introducing complexity at all levels of development and maintenance, from schema design to application code; not to mention the performance hit.
In 2014 we have access to many database servers that handle specific types of data exceptionally well.
Mongo (document storage)
Redis (key-value data storage)
MySQL/Maria/PostgreSQL/Oracle/etc (relational data)
CouchDB (JSON)
I'm sure I missed some others, like RabbirMQ and Cassandra. My point is, use the right tool for the data you need to store.
If your application requires storage and retrieval of a variety of data really, really fast, (and who doesn't) don't shy away from using multiple data sources for an application. Most popular web frameworks provide support for multiple data sources (Rails, Django, Grails, Cake, Zend, etc). This strategy limits the complexity to one specific area of the application, the ORM or the application's data source interface.
Early support for storing JSON in MySQL has been added to the MySQL 5.7.7 JSON labs release (linux binaries, source)! The release seems to have grown from a series of JSON-related user-defined functions made public back in 2013.
This nascent native JSON support seems to be heading in a very positive direction, including JSON validation on INSERT, an optimized binary storage format including a lookup table in the preamble that allows the JSN_EXTRACT function to perform binary lookups rather than parsing on every access. There is also a whole raft of new functions for handling and querying specific JSON datatypes:
CREATE TABLE users (id INT, preferences JSON);
INSERT INTO users VALUES (1, JSN_OBJECT('showSideBar', true, 'fontSize', 12));
SELECT JSN_EXTRACT(preferences, '$.showSideBar') from users;
+--------------------------------------------------+
| id | JSN_EXTRACT(preferences, '$.showSideBar') |
+--------------------------------------------------+
| 1 | true |
+--------------------------------------------------+
IMHO, the above is a great use case for this new functionality; many SQL databases already have a user table and, rather than making endless schema changes to accommodate an evolving set of user preferences, having a single JSON column a single JOIN away is perfect. Especially as it's unlikely that it would ever need to be queried for individual items.
While it's still early days, the MySQL server team are doing a great job of communicating the changes on the blog.
JSON is a valid datatype in PostgreSQL database as well. However, MySQL database has not officially supported JSON yet. But it's baking: http://mysqlserverteam.com/json-labs-release-native-json-data-type-and-binary-format/
I also agree that there are many valid cases that some data is better be serialized to a string in a database. The primary reason might be when it's not regularly queried, and when it's own schema might change - you don't want to change the database schema corresponding to that. The second reason is when the serialized string is directly from external sources, you may not want to parse all of them and feed in the database at any cost until you use any. So I'll be waiting for the new MySQL release to support JSON since it'll be easier for switching between different database then.
I know this is really late but I did have a similar situation where I used a hybrid approach of maintaining RDBMS standards of normalizing tables upto a point and then storing data in JSON as text value beyond that point. So for example I store data in 4 tables following RDBMS rules of normalization. However in the 4th table to accomodate dynamic schema I store data in JSON format. Every time I want to retrieve data I retrieve the JSON data, parse it and display it in Java. This has worked for me so far and to ensure that I am still able to index the fields I transform to json data in the table to a normalized manner using an ETL. This ensures that while the user is working on the application he faces minimal lag and the fields are transformed to a RDBMS friendly format for data analysis etc. I see this approach working well and believe that given MYSQL (5.7+) also allows parsing of JSON this approach gives you the benefits of both RDBMS and NOSQL databases.
I use json to record anything for a project, I use three tables in fact ! one for the data in json, one for the index of each metadata of the json structure (each meta is encoded by an unique id), and one for the session user, that's all.
The benchmark cannot be quantified at this early state of code, but for exemple I was user views (inner join with index) to get a category (or anything, as user, ...), and it was very slow (very very slow, used view in mysql is not the good way).
The search module, in this structure, can do anything I want, but, I think mongodb will be more efficient in this concept of full json data record.
For my exemple, I user views to create tree of category, and breadcrumb, my god ! so many query to do ! apache itself gone ! and, in fact, for this little website, I use know a php who generate tree and breadcrumb, the extraction of the datas is done by the search module (who use only index), the data table is used only for update.
If I want, I can destroy the all indexes, and regenerate it with each data, and do the reverse work to, like, destroy all the data (json) and regenerate it only with the index table.
My project is young, running under php and mysql, but, sometime I thing using node js and mongodb will be more efficient for this project.
Use json if you think you can do, just for do it, because you can ! and, forget it if it was a mistake; try by make good or bad choice, but try !
Low
a french user
I believe that storing JSON in a mysql database does in fact defeat the purpose of using RDBMS as it is intended to be used. I would not use it in any data that would be manipulated at some point or reported on, since it not only adds complexity but also could easily impact performance depending on how it is used.
However, I was curious if anyone else thought of a possible reason to actually do this. I was thinking to make an exception for logging purposes. In my case, I want to log requests that have a variable amount of parameters and errors. In this situation, I want to use tables for the type of requests, and the requests themselves with a JSON string of different values that were obtained.
In the above situation, the requests are logged and never manipulated or indexed within the JSON string field. HOWEVER, in a more complex environment, I would probably try to use something that has more of an intention for this type of data and store it with that system. As others have said, it really depends on what you are trying to accomplish, but following standards always helps longevity and reliability!
You can use this gist: https://gist.github.com/AminaG/33d90cb99c26298c48f670b8ffac39c3
After installing it to the server (just need root privilege not super), you can do something like this:
select extract_json_value('{"a":["a","2"]}','(/a)')
It will return
a 2
.You can return anything inside JSON by using this
The good part is that it is support MySQL 5.1,5.2,5.6. And you do not need to install any binary on the server.
Based on old project common-schema, but it is still working today
https://code.google.com/archive/p/common-schema/
I'm developing a form generator, and wondering if it would be bad mojo to store JSON in an SQL database?
I want to keep my database & tables simple, so I was going to have
`pKey, formTitle, formJSON`
on a table, and then store
{["firstName":{"required":"true","type":"text"},"lastName":{"required":"true","type":"text"}}
in formJSON.
Any input is appreciated.
I use JSON extensively in my CMS (which hosts about 110 sites) and I find the speed of access data to be very fast. I was surprised that there wasn't more speed degradation. Every object in the CMS (Page, Layout, List, Topic, etc) has an NVARCHAR(MAX) column called JSONConfiguration. My ORM tool knows to look for that column and reconstitute it as an object if needed. Or, depending on the situation, I will just pass it to the client for jQuery or Ext JS to process.
As for readability / maintainability of my code, you might say it's improved because I now have classes that represent a lot of the JSON objects stored in the DB.
I used JSON.net for all serialization / deserialization. https://www.newtonsoft.com/json
I also use a single query to return meta-JSON with the actual data. As in the case of Ext JS, I have queries that return both the structure of the Ext JS object as well as the data the object will need. This cuts out one post back / SQL round trip.
I was also surprised at how fast the code was to parse a list of JSON objects and map them into a DataTable object that I then handed to a GridView.
The only downside I've seen to using JSON is indexing. If you have a property of the JSON you need to search, then you have to store it as a separate column.
There are JSON DB's out there that might server your needs better: CouchDB, MongoDB, and Cassandra.
A brilliant way to make an object database from sql server. I do this for all config objects and everything else that doesn't need any specific querying. extending your object - easy, just create a new property in your class and init with default value. Don't need a property any more? Just delete it in the class. Easy roll out, easy upgrade. Not suitable for all objects, but if you extract any prop you need to index on - keep using it. Very modern way of using sql server.
It will be slower than having the form defined in code, but one extra query shouldn't cause you much harm. (Just don't let 1 extra query become 10 extra queries!)
Edit: If you are selecting the row by formTitle instead of pKey (I would, because then your code will be more readable), put an index on formTitle
We have used a modified version of XML for exactly the purpose you decribe for seven or eight years and it works great. Our customers' form needs are so diverse that we could never keep up with a table/column approach. We are too far down the XML road to change very easily but I think JSON would work as well and maybe evan better.
Reporting is no problem with a couple of good parsing functions and I would defy anyone to find a significant difference in performance between our reporting/analytics and a table/column solution to this need.
I wouldn't recommend it.
If you ever want to do any reporting or query based on these values in the future it's going to make your life a lot harder than having a few extra tables/columns.
Why are you avoiding making new tables? I say if your application requires them go ahead and add them in... Also if someone has to go through your code/db later it's probably going to be harder for them to figure out what you had going on (depending on what kind of documentation you have).
You should be able to use SisoDb for this. http://sisodb.com
I think it not an optimal idea to store object data in a string in SQL. You have to do transformation outside of SQL in order to parse it. That presents a performance issue and you lose the leverage of using SQL native data parsing capability. A better way would be to store JSON as an XML datatype in SQL. This way, you kill two birds with one stone: You don't have to create shit load of tables and still get all the native querying benefits of SQL.
XML in SQL Server 2005? Better than JSON in Varchar?