Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I have CSV File's With Different Columns With Few Common Columns, We Are Currently Using Excel To Remove Unwanted Rows Clean The Data, and Generate Reports, I Am Thinking Of Using Elasticsearch As A Solution For Data Storage, Transformation, Load And Reporting.
Is Elasticsearch A Good Choice For This Use Case ?
Elastic Search is, as the name indicated, using to quick search. It is build upon Lucene and similar to another Apache project, Solr...
If you want to query the raw data or do some simple aggregation upon it. It is fine and you can also use Kibana to come up with some fancy GUI so your audience can interact with the data and you can even come up with some dashboard to demonstrate some basic staff. However, it is not a replacement of a data base.
If you want to update or join.. you had better use some data base ... sql + mongo or hive for big data.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
Welcome, all
I would like to create a small web application, which needs to scrape data from another website and save the results to a database.
So, currently, I have Laravel 8 installed on my local windows machine, and I made a connection to the database, where I have "items" in the table that I need to save the data in
And the data may be changed from another website so I need this change in the database as well,
and need to scrape many items from another website so I need to know id or something else to distinguish one item from another
Now, the question is, what is the best way to do this? I've tried some tutorials but I haven't.
You can use a package Goutte or domcrawler, first you need to do a request of a url for that you can use GuzzleHttp or get the portion of html content and filter it and then save all this data on a table of your database.
You can read more information about those libraries.
Another library that is more customizable is domcrawler Dom_Crawler,
Goutte
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I'm building an application with the instant messaging functionality.
The application will allow the users to send files/images as well as normal text messages.
I decided to take the approach with storing the files on the filesystem and write only the file paths to the database. There will be no updates to the files (only insertions and deletions).
Which database would be the best for storing a large amount of file paths, that would be easy to query for a certain user files?
I would go with MongoDB. My experience is that a document based approach using a single Messages collection would be best. Each message document then contains all of the file paths. This eliminates joins and better supports potential future functional requirements changes.
MongoDB also provides great ways to deal with old messages such as TTL indexes.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I'm working on a web project in clojure and I will have a mysql database storing information. I plan on using Korma for data CRUD but I'm trying to figure out the best way to manage actually creating the tables.
I'm inclined to just write a bash script that runs the CREATE TABLE commands, but I feel like this isn't the proper way to do things, and doesn't allow for future migrations. Should I use a Clojure library for this? Bash? Something else entirely?
Do you mean this one Migratus, this is used in luminus(A web framework), And I think this might help.
Liquibase is a popular tool for database migrations in the Java world. There is also a clj-liquibase project which uses EDN and uses liquibase internally, but I've never used it.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I want to make a simple script that periodically performs SNMP GET and stores the output to MySQL database. It will be running on Windows.
What would be the least painful choice of programming language to write it in? Powershell, Java, …?
Python has SNMP (i.e. NET-SNMP) bindings. It will handle MySQL database as well. In that way you may use those APIs to create your own scanning application with SNMP communication channel and MySQL database storage of collected data. Of course there are tons of other solutions/combinations possible... however I recommend this one, because in such a simple case you don't have to engage heavy JVM.
Python has SNMP(Net-SNMP) binding and it's better that you can use RRD database to store output as it consists time interval itself.Python has RRD api also and you can handle easily
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I am trying to write a webapp, where one of the functionality is to exchange messages. I am trying to understand how to store these messages. I do not want to store it in DB. If i have to store in file, then how do i separate between messages.
Any links to some document would be greatly appreciated. I tried googling a lot but could not get hold of any reference
You should think about storing the messages in XML format, and use your webapp to load and parse those XML files into the message objects. Why do you not want to store the messages in the database? There are serious drawbacks to storing in the file system rather then the database (or even system memory).
A file system is a database, just not a relational database.
It's often faster than a relation database, but it has significantly less flexibility for indexing on multiple fields.
Parsing XML is gonna suck whether the XML comes from a database or a file.
Instead, you should do page caching to the file system of HTML, or HTML fragments.