Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I'm trying to retrieve a couple of webpages so that I can manipulate their information. I am looking for what are the best tools and packages for doing this. Preferably, I would appreciate to use Perl for parsing the tables, but I would be fine with using MySQL too, though I would have to learn it.
(I believe they are HTML tables, despite the page being coded in Java?).
I am not asking for any code, but for your opinion, tips and suggestions on how I can accomplish this. My intentions are, for now, to just parse the tables, merge them, and build a single table that has the information from both sites in different columns. This general table would only need to be updated once a day at most.
The tables can be found in here, please scroll down and in here, the full table.
Thank you in advance for all the help!
For web page retrieve, you can use the Web::Scraper, HTML::DOM, and also the LWP related modules.
And for SQL, the DBI module is very powerful. You can search these using the cpan.
Your question seems to be how to do webscraping. I don't see how MySQL would come into play with this. For webscrapint I suggest that you look at the CPAN modules WWW::Mechanize and HTML::TableExtractor
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
as the final project for a programming course I took I've made a small tool which deals with databases issues. It works and I think it could be useful to others, so I'd like to put the code on the net to see if it actually is.
I'm aware of various source code hosts such as GitHub or Google Code, however they seem more oriented towards group development and I'm not sure if they're good for individual complete projects as well. I'm not saying I'm opposed to collaboration, but those platforms just seem a bit too much for something so simple.
Does anyone have any idea on how I should share the code?
Github would be fine - just because git allows multiple developers on a project doesn't mean it is required.
Most of the complexity is there if you need it but you do not have to ever visit those screens. The last time I looked at the network view for my more complex projects was a long time ago!
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
There is a site that I would like to replicate i would like to know how there database is structured. Is there a tool that can help with this process?
That's not possible because the web pages do not have (direct) information on the underlying data dictionary. They have HTML, JS, CSS. They have no data model not even SQL queries.
You can look at the HTML/JS code which might give you some ideas, that indirectly, lead you to a possible design. This is the work any professional database designer should be able to do.
You would need to ask the web site owners what is the data dictionary or use some form of cracking (illegal in most countries, I guess) to have access to the server/dbms.
But even if you could have access to the data dictionary it is not necessarily obvious how that translates into the web pages you are vieweing. That's because usually there is data transformation (business rules) to arrive to the web pages, usually through some form of templating.
What you need is reverse engineering, and for that the main tool is the brain. You could use some scripting to extract useful data, eg, label elements or table headings.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I was hoping someone could point me in the right direction. I have moved from a mysql db to mongodb. Is there any tools available to run against the two databases to ensure that the data is consistent.
Ive asked Google but ive been coming up empty. The other option i was considering was to write a script and take random samples from the db's and compare the results.
I doubt there is anything for this. The only way that I can see would be to write some script that compares the two data sources. But I do have to point out that if you just merely moved tables to collections, you will run into problems. The data schema that MongoDB likes for optimal performance is in no means going to be similar to what you'd use in a relational database such as MySQL.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I have a postgresql dump file and wants to convert it to MySQL. I google it and found pg2mysql which it converted successfully. However, I get errors when importing it to mysql. Looks like the formats are not compatible or wrong.
Any one can recommend?
Thanks.
James
The best solution for a migration would be a conversion tool like ESF data migration toolkit or openDBcopy that is open source.
According to your pastebin, the problem is with bigserial, as this is not a MySQL datatype. Some Googling came across this site which describes what bigserial is in Postgres context and other RDBMS datatype equivalents.
Looks like there are 3rd party libraries which will convert the data from Postgres to whichever RDMBS you choose, an example is easyfrom.net, if this doesn't fit your needs, just google around, as you are not the first person to need this functionality.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 8 years ago.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
This question does not appear to be about programming within the scope defined in the help center.
Improve this question
I'm running an open source project and I'm wanting to externalize development discussions that we have been having by email, make them public and allow anyone to take part in the conversation. For this, I've been thinking about setting up a mailing list.
Given this, is a mailing list currently the be "device" for this. If so what do people think is the best tool to use for this? Is Google Groups still the best tool for mailing lists, is there something people that OSS is gravitating more towards? Surely there is something better, Google Groups seems outdated.
GNU Mailman is a very popular package used on mailing list servers; you may want to use this to manage and process your list.
Maybe Librelist? A number of open source projects is using it such as Flask. Librelist.com