I have a lot of text data broken down into articles and I anticipate once I release this data a lot of people will be viewing it.
So do I let each person query a MySQL database and house the data there or should I use XML and have one article per xml file and just parse it on the fly?
I am using PHP, MySQL.
I think that MySQL would be a lot faster?
What about some other format to store all this data in?
All things considered, you should look into NoSQL and document oriented databases.
As always, there's more than one way to kin a cat.
One way would be to store the articles into a database and use full text indexes.
You could consider reformatting the data into HTML (that is stored into a structures of files and directories) and index the articles using nutch, or solr.
Related
I'm a newbie web developer.
I am trying to make my own web based text editor application thus I need this info.
How do I store text with line breaks in database.
I am thinking to use MongoDB.
Also, I'm interested in knowing:
What database YouTube uses to store comments?
What database stackoverflow uses to store comments?
Thanks in Advance.
When deciding what type of database to use for your own project, it is more important to keep in mind what features of that database that you may need for your own project.
YouTube and Stackoverflow both use SQL databases, but the result that you are trying to accomplish can be performed in a NoSQL database like MongoDB as well.
If you have a fixed schema and need to perform table joins in order to compile all of your data, a SQL database might be the way to go.
If your schema is more flexible and your data would benefit from document-based storage and loading, then a NoSQL database is the way to go.
I have JSON file that I want to save as a blob to Microsoft SQL Server.
The pros for zipping is saving space, the cons is the readability that getting lost.
I want to know if T-SQL has any optimization in which it zips the blobs on its own. I know that columnar databases work this way, like Vertica or Postgres for example.
I personally would not compress them if I wanted to be able to search by them. I do not believe it compresses a blob on it's own. I know for a fact even just very large VARCHAR columns do not compress on their own, so I would not expect a blob to. However there is built in compression you can turn on:
https://blogs.msdn.microsoft.com/sqlserverstorageengine/2015/12/08/built-in-functions-for-compressiondecompression-in-sql-server-2016/
https://learn.microsoft.com/en-us/sql/relational-databases/data-compression/enable-compression-on-a-table-or-index?view=sql-server-2017
There are some advantages to it but usually at the cost of CPU. So if I were you, I'd probably not zip up the files to put in SQL, but I might compress the tables I store. It would depend on exactly what the data was, json probably gets a lot of space back on compression, but a .jpeg would not.
An option I have done in the past is to simply store my files on a content server somewhere, and store in SQL the meta data about the file (name, tags, patch to where I stored it, file extension, etc.) That way my data is easy to get at/put there and I simply use SQL to look it up. Additionally it has allowed me when it was large text files to also use Lucene indexes from solr to make a full text searchable solution since the data wasn't stuffed into a SQL table. Just an idea! :)
One more thought, if I were to store big json files into SQL I would probably choose VARCHAR(MAX) or NVARCHAR(MAX) as my datatype. Anytime I have tried to use TEXT, IMAGE, etc. I would later run into some kind of SQL error if I tried to do a tricky query. I believe Microsoft is trying to use VARCHAR(MAX) to replace the blob type of data types and is slowly deprecating them.
I am using MongoDB first time here. Is there any best way to transfer bulkdata from mysql into MongoDB. I try to search to in different ways but i did not find.
You would have to physically map all your mysql tables to documents in mongodb but you can use an already developed tool.
You can try: Mongify (http://mongify.com/)
It's a super simple way to transform your data from a MySql to MongoDB. It has a ton of support for changing your existing schema into a schema that would work better with MongoDB.
Mongify will read your mysql database, build a translation file for you and all you have to do is map how you want your data transformed.
It supports:
Updating internal IDs (to BSON ObjectID)
Updating referencing IDs
Typecasting values
Embedding Tables into other documents
Before filters (to change data manually before import)
and much much more...
There is also a short 5 min video on the homepage that shows you how easy it is.
Try it out and tell me what you think.
Please up vote if you find this answer helpful.
I need help from you experts about practices regarding database development. I have a few questions regarding MySQL databases:
Is there a way for MySQL that a database and its structure is developed in an XML language and then converted to a fully functional MySQL database?
Is it possible to generate the XML source file from question 1 (see above) based on an existing database in MySQL ?
As far as I know, XML is not suitable for developing database structures. However can we say that XML is a language to demonstrate hierarchical structures and a MySQL database also shows a hierarchical structure, so in fact it is suitable for database development?
Thank you very much!
You can certainly store XML data in MySQL. You can also use any number of approaches to converted hierachical XML data into individual relational database field representations.
I would however say that if you just want to work with intact XML documents, you might look to go the NoSQL route, which is really better suited for this type of data storage. You also might consider JSON as the format for storage as it is more concise (saves space and transmissions badnwidth) and is more aligned with the popular NoSQL data stores out there.
1) yeah there is a way, but you should check out mongodb if you want a dynamic database structure, it was developed with that in mind. also, unless you need the rss features of xml or something similar, you might want to consider using json as a format for you documents.
2) json and mongodb work very well together to quickly and easily get documents in and out of the db. you can technically do it in mysql as well, but you might spend more time scripting in php or ruby to get the desired format you want.
3) you could use xml to demonstrate your db structure because of it's loose structure, but i'm not sure it would be intuitively clear to others. hard to say, really depends on how you implement it and how complicated your db structure is going to be.
I have a few XML files containing data for a research project which I need to run some statistics on. The amount of data is close to 100GB.
The structure is not so complex (could be mapped to perhaps 10 tables in a relational model), and given the nature of the problem, this data will never be updated again, I only need it available in a place where it's easy to run queries on.
I've read about XML databases, and the possibility of running XPATH-style queries on it, but I never used them and I'm not so comfortable with it. Having the data in a relational database would be my preferred choice.
So, I'm looking for a way to covert the data stored in XML into a relational database (think of a big .sql file similar to the one generated by mysqldump, but anything else would do).
The ultimate goal is to be able to run SQL queries for crunching the data.
After some research I'm almost convinced I have to write it on my own.
But I feel this is a common problem, and therefore there should be a tool which already does that.
So, do you know of any tool that would transform XML data into a relational database?
PS1:
My idea would be something like (it can work differently, but just to make sure you get my point):
Analyse the data structure (based on the XML themselves, or on a XSD)
Build the relational database (tables, keys) based on that structure
Generate SQL statements to create the database
Generate SQL statements to create fill in the data
PS2:
I've seen some posts here in SO but still I couldn't find a solution.
Microsoft's "Xml Bulk Load" tool seems to do something in that direction, but I don't have a MS SQL Server.
Databases are not the only way to search data. I can highly recommend Apache Solr
Strategies to Implement search on XML file
Keep your raw data as XML and search it using the Solr index
Importing XML files of the right format into a MySql database is easy:
https://dev.mysql.com/doc/refman/5.6/en/load-xml.html
This means, you typically have to transform your XML data into that kind of format. How you do this depends on the complexity of the transformation, what programming languages you know, and if you want to use XSLT (which is most probably a good idea).
From your former answers it seems you know Python, so http://xmlsoft.org/XSLT/python.html may be the right thing for you to start with.
Take a look at StAX instead of XSD for analyzing/extraction of data. It's stream based and can deal with huge XML files.
If you feel comfortable with Perl, I've had pretty good luck with XML::Twig module for processing really big XML files.
Basically, all you need is to setup few twig handlers and import your data into MySQL using DBI/DBD::mysql.
There is pretty good example on xmltwig.org.
If you comfortable with commercial products, you might want to have a look at Data Wizard for MySQL by the SQL Maestro Group.
This application is targeted especially at exporting and, of course, importing data from/ to MySQL databases. This also includes XML import. You can download a 30-day trial to check if this is what you are looking for.
I have to admit that I did not use the MySQL product line from them yet, but I had a good user experience with their Firebird Maestro and SQLite Maestro products.