I want to store file data of a directory in a file. i.e., file name, file size etc so that I can reduce search time. My problem now is to find an efficient way to do it. I have considered json and xml but can't decide between these two. Also if there is a better way let me know.
I'd say that it's up to what kind of data you prefer to work with and to what structure of data you have (very simple as a list of word, less simple as a list of word and the number of time each word was searched,...)
For a list of word you can use a simple text file with one word per line or coma separated (csv), for a less simple structure, json or xml will work fine.
I like to work with json as it's more light than xml and less verbose. If you didn't plan to share this data and/or it isn't complex, you don't need the validation (xsd,...) offered by xml.
And even if you plan to share this data, you can work with json.
You'll need some server side code to write the data to a file, like php, java, python, ruby,...
I would recommend Json file if you use alomst like a properties file.
If you plan to store the data in the file into database then you can go for XML where u have to the option to use JAXB/JPA in java environment
Related
If I have understood this correctly, one advantage of a document-oriented database like MongoDB is that unstructured data can be stored well. For example, If I have different HTML files, is the complete storage in a string in each document an advantage ? Or is it so thought that the different single contents of the HTML files are stored separately as many key-value pairs and only then make a HTML file out of it ?
If its a matter of storing the HTML content only, you can directly save it as a string. Text searching can also be done easily by going this route.
On the other side, if you need to process different elements, dynamically assign some values, replace some placeholders, etc. then you can think of saving associated tags/elements separately.
I am designing a system with 30,000 objects or so and can't decide between the two: either have a JSON file pre computed for each one and get data by pointing to URL of the file (I think Twitter does something similar) or have a PHP/Perl/whatever else script that will produce JSON object on the fly when requested, from let's say database, and send it back. Is one more suited for than another? I guess if it takes a long time to generate the JSON data it is better to have already done JSON files. What if generating is as quick as accessing a database? Although I suppose one has a dedicated table in the database specifically for that. Data doesn't change very often so updating is not a constant thing. In that respect the data is static for all intense and purposes.
Anyways, any thought would be much appreciated!
Alex
You might want to try MongoDB which retrieves the objects as JSON and is highly scalable and easy to setup.
I am aware of the batch LOAD XML technique e.g. Load XML Update Table--MySQL
Can MySql insert/replace rows directly from xml. I'd like to pass an XML string to MySQL.
Something like replace into user XML VALUES maybe even using as to map the tags to the column names??
The primary thing is that I dont want to parse the XML in my code, I'd like MySql to handle this. I dont have a file, I have the XML as a string.
I have looked and found there are some XML Functions:
12.11. XML Functions
The XML functions can do XPath, but I think this is a little fiddly as I have a 1:1 mapping from the XML to the table structure so I'd hjst like to be able to say hey MySql, insert the values in the xml string in to the table.
Is this possible?
In a nutshell, No.
What your looking for is an XML storage engine for MySQL. There has never been one created officially, and i have never seen a third party one either (but feel free to google).
If you really want to achieve this, then the closest you would get is to look for an alternative (R)DMS, but then that might not support the type of queries you wish to perform, may require a bit of a learning curve, would no doubt require you are using a server with superuser access, and potentially mean re-factoring a lot of your code.
I have access to an web service that returns an XML or JSON. Since web service calls are expensive, I want to store the XML/JSON in my database so that I can access the data faster. The question I have is if I should just store the entire XML/JSON in a field or should I design a database model that represents the XML/JSON in a normalized way?
If I just needed to have the XML/JSON data available to me, then saving it as a string in a field would be OK.
However, I know that I'll be needed to extract only certain XML/JSON documents -- so I kindda need to be able to query this. For simple queries, maybe I can use something like LIKE %<title>hello world</title>% if I was search for "hello world" between the title tags within an XML document. But I think some of my queries might go beyond string matching (e.g. greater than a certain number or date, etc.). So I feel like I need to model this data properly in my database and populate it with the values from XML/JSON. This will become a painful exercise, though.
Any tips on what I should do? Maybe there is an option I didn't consider.
It really sounds like you have the requirement to translate the XML/JSON document into a distinct set of fields.
There are DBs out there that can return native JSON, but you normally would use the application layer.
I normally store pure XML/JSON when there's really no obvious need to access individual fields. For example, I'll store control data in XML format in a BLOB field, since I don't generally need to search for one particular control string.
One thing that you might want to think about is using a noSQL solution like MongoDB or Couch to store the JSON. You can put the JSON string and pull it out directly as well as access the individual fields.
A document database like MongoDB and Couch gives you the flexibility to natively store and retrieve the JSON document as well as access and search on the individual fields inside the JSON document, without needing to in advance know how the document is going to be used. It is really just eliminating a couple of steps in converting relational data into a structured document and converting a structured document into relational data.
When I do store XML directly in a BLOB, any of the searchable data would be in a field outside the BLOB. Disk space is relatively cheap, and this is a minor de-normalization. Sure, you'll have to make sure to to keep the field updated whenever the JSON/XML document is updated, but that's easy to do at the application layer.
I'm a real beginner when it comes time for this, so I apologize in advance.
The long and short of what I am looking for is a fairly simple concept - I want to pull JSON data off a server, parse it, and load it into excel, access, or some other type of tables. Basically, I want to be able to store the data so I can filter, sort, and query it.
To make matters a little more complicated, the server will only return truncated results with each JSON, so it will be necessary to make multiple requests to the server.
Are there tools out there or code available which will help me do what I am looking for? I am completely lost, and I have no idea where to start.
(please be gentle)
I'm glad seeing this question b/c I'm doing very similar things! And based on what I'd gone through, it has lot to do with how those tables are designed or even linked together at first, and then the mapping between these tables and different JSON objects at different depth or position in the original JSON file. After the mapping rules are made clear, the code can be done by merely hard-coding the mapping(I mean like: if you got JSON object after a certain parent of it, then you save the data into certain table(s)) if you're using some high level JSON paring library.
OK as i have to dash home from the office now:
Assuming that you are going to use Excel to Parse the data you are going to need:
1.Some Json Parser JSON Parser for VBA
2.Some code to download the JSON
3.A loop of VBA code that loops through each file and parses it into a sheet.
Is this ok for a starter? If you are struggling let me know and I will try and knock something up a little better over the weekend.