I'm after a bit of advice here if possible.
I basically have 12 XML files which I need to use in my mysql db. These xml files are all in a completely different structure and the data changes constantly.
With this in mind what would be the best approach for bringing this in and using it and updating it. I had thought to use a cronjob to execute a php file to write each of these into thier own table, but baring in mind there are 12 files, with around 60 lines in each file and the cron job will need to run every 15 minutes I think this will end up killing the server.
Any ideas on a solution would be gratefully appreciated.
Thanks
Richard
You can use the LOAD_XML command in mysql to do this:
http://dev.mysql.com/doc/refman/5.5/en/load-xml.html
If the data is changed constantly in the XML files, how often does it need to be up to date in the database? If the database is just a backing record for the XML, then maybe you can update once or twice an hour.
Related
I am using SSIS to move data between a local MSSQL server table to a remote MYSQL table (Data flow, OLEdb source and ODBC Destination). this works fine if im only moving 2 lines of data, but is very slow when using the table I want which has 5000 rows that fits into a csv of about 3mb, this currently takes about 3 minutes using ssis's options, however performing the steps below can be done in 5 seconds max).
I can export the data to a csv file copy it to the remote server then run a script to import straight to the DB, but this requires a lot more steps that I would like as I have multiple tables I wish to perform the steps on.
I have tried row by row and batch processing but both are very slow in comparison.
I know I can use the above steps but I like using the SSIS GUI and would have thought there was a better way of tackling this.
I have googled away multiple times but have not found anything that fits the bill so am calling on external opinions.
I understand SSIS has its limitations but I would hope there is is a better and faster way of achieving what I am trying to do. If SSIS is so bad I may as well just rewrite everything into a script and be done with it, But I like the look and feel of the Gui and would like to move my data in this nice friendly way of seeing things happen.
any suggestions or opinions would be appreciated.
thank you for your time.
As above have tried ssis options including a 3rd party option cozyroc but that sent some data with errors (delimiting on columns seemed off) now and again, different amount of rows being copied and enough problems to make me not trust the data.
I need daily load the result of a specific query to Redshift. I've already created a table on redshift that will support the results of this query but now i'm a little stuck since i can't find a good way to solve this.
So far i tried using python but im getting lots of headaches regarding line terminators in fields that basically store a description, and character encodings.
I know lots of programs that allow you to connect to a db and make querys also have an export option to csv but since i need to automatically do this everyday i don't think any of those would work for me.
Now i would like to know if maybe there are better suited options so i can start looking into them. Im not asking for a step by step how to but just for tools/programs/etc that i should start looking into.
You should look into MySQL's stored procedures and events -- using just MySQL, you can have it generate a file every day.
You can't dynamically rename the file, or overwrite it, though, so you'd need a second job which deletes the file -- this can be done with Python.
Whether you're running Windows or Linux, you should be able to schedule a batch file or python script to execute once a day, and that would be an alternate way to do that.
Does this address your question?
I have an Access database that keeps track of many different aspects of my companies performance and I would like to add functionality to keep track of the hours the employees are working.
The hours are all kept track of on a website called timetracker. They have a few reporting options including XML and CSV files. The site has a favorite report feature to get the same data in the format that I want it every week.
What I would like to do is find the best process for getting the data from this website, into a table in my database that I can reference.
I will not be the one executing whatever process I come up with and I would really like it to be as easy as possible for whoever it is that does have to do it.
Right now I have a linked table that is an XML file in our SharePoint folder. I was thinking that maybe we could just run the report and download the file every week then just save it over the old file with the correct sheet names and it should update.
What I am wondering is if anyone can come up with an easier process for doing this that would take the least amount of time and be easiest to write down instructions for that anyone could execute.
(Would it maybe be possible to create some sort of macro to actually download the report automatically?)
The want a Perl script that will write to a data file every time update the Database MySQL. I dont mind about growth of the file since Every Items audited will be stored seperatly
Thank you will Appreciate you Help
The module Log::Log4perl provides many different ways to log events to many types of output including files. It also would allow you to set debug levels to turn this off if you needed to.
I have inherited a legacy Access app that has a TON of code running before the main form comes up. This code imports data, deletes data, and changes data a million ways.
Is there a way where (after the startup stuff is finished) I can list the tables and when each of them had data affected most recently?
Thanks!
Sorry, but I'm afraid the answer to your question is simply: No.
Here's an idea though:
Make a backup of the database file.
Open the app so it runs the code you are concerned about.
Compare the DB to the backup using a tool like Red Gate SQL Data compare.
BTW: I'm not sure if the RedGate tool works against access databases, but FMSInc. has one that claims to.