I am designing a system with 30,000 objects or so and can't decide between the two: either have a JSON file pre computed for each one and get data by pointing to URL of the file (I think Twitter does something similar) or have a PHP/Perl/whatever else script that will produce JSON object on the fly when requested, from let's say database, and send it back. Is one more suited for than another? I guess if it takes a long time to generate the JSON data it is better to have already done JSON files. What if generating is as quick as accessing a database? Although I suppose one has a dedicated table in the database specifically for that. Data doesn't change very often so updating is not a constant thing. In that respect the data is static for all intense and purposes.
Anyways, any thought would be much appreciated!
Alex
You might want to try MongoDB which retrieves the objects as JSON and is highly scalable and easy to setup.
Related
So this is more of a conceptual question. There might be some fundamental concepts which I don't understand clearly so please point out any mistakes in my understanding.
I am tasked with designing a framework and a part of it is I have a MySQL DB and a REST API which acts as the Data Access Layer. Now, the user should be able to parse various data (JSON, CSV, XML, Text, Source Code etc.) and send it to the REST API which persists the data to the DB.
Question 1: Should I specify that all data sent to the REST API should be in JSON format no matter what is parsed? This will ensure (best to my understanding) language independence and gives the REST API a common format to deal with.
Question 2: When it comes to a data model, what should I specify? Is it like a one-model-fits-all sort of thing or is the data model subject to change based on the incoming data?
Question 3: When I think of a relational data model, the thought of foreign keys comes to mind which creates the relation. Now, it might happen that some data may not contain any relation at all. If we think of something like Customer Order sort of data then the relation is easy to identify. But what if the data does not have any relation at all? How does the relational model fit into this?
Any help/suggestion is greatly appreciated. Thank you!
EDIT:
First off, the data can be both structured (say XML) and unstructured (say two text files). I want the DAL to be able to handle and persist whatever data that comes in (that's why I thought of a REST interface in front of the DB).
Secondly, I also just recently thought about MongoDB as an option and was looking into it (I have never used NoSQL DBs before). It kind of makes sense to use it if the incoming data in REST is in JSON. From what I understood I can create a collection in Mongo. Does that make more sense than using a Relational DB??
Finally, as to what I want to do with the data is I have a tool which performs a sort of difference analysis (think git diff) on the data. Say I sent two XML files and the tool retrieves it from the DB and performs the difference analysis and stores the result back in the DB.
Based on these requirements, what would be the optimum way to go about it?
The answer to this will depend on what sort of data it is. Are all these different data types using different notation for the same data? If so then storing in normalised database tables is the way to go. If its just arbitrary strings that happen to have some form of encoding, then its probably best to store in raw.
Again, it depends on what you want to do with it afterwards. Are you analysing the data, and you reporting on it? Are you reading one format and converting to another? Is it all some form of key-value pairs in some notation or other
No way to answer this further without understanding what you are trying to achieve.
I've inherited some code which makes calls to a web API, and get's a deeply nested (up to eight levels) response.
I've written some code to flatten the structure so that it can be written to .csv files, and a SQL database, for people to consume more easily.
What I'd really like to do though is keep a version of the original response, so that there's a reference of the original structure if I ever want/need it.
I understand that HDF5 is primarily meant to store numerical data. Is there any reason not to use it to dump JSON blobs? It seems a lot easier than setting up a NoSQL database.
It should be fine. It sounds like you'd be storing each JSON response as a HDF5 variable length string. Which is fine, it's just a string to the library.
Do you plan to store each response as a separate dataset? That may be inefficient if you are talking about >1000's of responses.
Alternatively, you can create a 1-d extensible dataset, and just append to it with each response.
Decided it was easier to set up a Mongo database.
I'm sitting on a CSV import into a database trying to think of ways to add more data without having to change the API (add new fields).
Since working with JSON quite a bit on the client, I'm thinking of storing data into MYSQL as JSON string. So if I have a field
image_filenames
Where I'm currently storing data like this:
img123.jpg
WOuld it make sense to store multiple images in a JSON array like so:
{"img_base":"img123.jpg", "img_alt_1":"img123-1.jpg", "img_alt_2":"img123-2" }
I can deserialize server side, so it woudn't be much of a problem to grab the image I need from the JSON array, while it does not bloat up the API.
Question:
I can't find anything at all on importing CSV with JSON strings. So, what's good and bad in doing so? Are there security concerns (SQL-injections)?
Thanks!
Transferred from comments to here:
If you have a data model scheme that changes or is inconsistent, then the relational database storage isn't the best choice. Sure, you can serialize it and store as binary string, but why? IMO, and I'm not a fan of NoSQLs, but MongoDB looks like something you might make use of. Its document scheme is JSON-based, it'd be familiar to you if you work with JSON-based code on a daily basis. I'd use that to store the data rather than relational db.
Non-relational ones do less work, so they work faster in some scenarios. They also don't have a scheme, so there's no alter table statement as such, therefore you can add "columns" as much as you like. If you don't have relations and need something to store data in JSON format, but that it can be searchable - MongoDB would be great.
I have access to an web service that returns an XML or JSON. Since web service calls are expensive, I want to store the XML/JSON in my database so that I can access the data faster. The question I have is if I should just store the entire XML/JSON in a field or should I design a database model that represents the XML/JSON in a normalized way?
If I just needed to have the XML/JSON data available to me, then saving it as a string in a field would be OK.
However, I know that I'll be needed to extract only certain XML/JSON documents -- so I kindda need to be able to query this. For simple queries, maybe I can use something like LIKE %<title>hello world</title>% if I was search for "hello world" between the title tags within an XML document. But I think some of my queries might go beyond string matching (e.g. greater than a certain number or date, etc.). So I feel like I need to model this data properly in my database and populate it with the values from XML/JSON. This will become a painful exercise, though.
Any tips on what I should do? Maybe there is an option I didn't consider.
It really sounds like you have the requirement to translate the XML/JSON document into a distinct set of fields.
There are DBs out there that can return native JSON, but you normally would use the application layer.
I normally store pure XML/JSON when there's really no obvious need to access individual fields. For example, I'll store control data in XML format in a BLOB field, since I don't generally need to search for one particular control string.
One thing that you might want to think about is using a noSQL solution like MongoDB or Couch to store the JSON. You can put the JSON string and pull it out directly as well as access the individual fields.
A document database like MongoDB and Couch gives you the flexibility to natively store and retrieve the JSON document as well as access and search on the individual fields inside the JSON document, without needing to in advance know how the document is going to be used. It is really just eliminating a couple of steps in converting relational data into a structured document and converting a structured document into relational data.
When I do store XML directly in a BLOB, any of the searchable data would be in a field outside the BLOB. Disk space is relatively cheap, and this is a minor de-normalization. Sure, you'll have to make sure to to keep the field updated whenever the JSON/XML document is updated, but that's easy to do at the application layer.
I'm a real beginner when it comes time for this, so I apologize in advance.
The long and short of what I am looking for is a fairly simple concept - I want to pull JSON data off a server, parse it, and load it into excel, access, or some other type of tables. Basically, I want to be able to store the data so I can filter, sort, and query it.
To make matters a little more complicated, the server will only return truncated results with each JSON, so it will be necessary to make multiple requests to the server.
Are there tools out there or code available which will help me do what I am looking for? I am completely lost, and I have no idea where to start.
(please be gentle)
I'm glad seeing this question b/c I'm doing very similar things! And based on what I'd gone through, it has lot to do with how those tables are designed or even linked together at first, and then the mapping between these tables and different JSON objects at different depth or position in the original JSON file. After the mapping rules are made clear, the code can be done by merely hard-coding the mapping(I mean like: if you got JSON object after a certain parent of it, then you save the data into certain table(s)) if you're using some high level JSON paring library.
OK as i have to dash home from the office now:
Assuming that you are going to use Excel to Parse the data you are going to need:
1.Some Json Parser JSON Parser for VBA
2.Some code to download the JSON
3.A loop of VBA code that loops through each file and parses it into a sheet.
Is this ok for a starter? If you are struggling let me know and I will try and knock something up a little better over the weekend.