I’ve searched but can’t find exactly what I’m trying to do, or just losing my mind... probably both.
I’ve got a data feed, with the following fields: Product ID, Name, Retail Price, Sales Price. Right now I dump this into a Google Sheets file via scripts, trim the necessary fields, export via CSV and back in MySQL but I need the data to go right from the feed CSV URL to MySQL, and correct a few fields.
For some reason, some of the prices have weird values, such as 4728.7376 while others have 282. I’d like to basically trim andything past the decimal point. When I tried that, I have some rows that don’t have any prices (it’s annoying) and seems to break the import.
Any suggestions on simplifying this to pull from the URL feed, fix issues and import into MySQL — which is through Google Cloud, but currently testing using MAMP. I can only get it to work importing using varchar(100) and need to do calculations with those columns; as long as the value is present.
Thanks in advance!
Related
On our Wordpress site, we use a plugin called s2member and it stores the levels (roles) of our clients as well as the times they were assigned a specific level in our database. I would like to create a table that shows when a user was assigned a specific level. I'm having a challenge getting the data I need because of the way the data is stored in the field. It stores all of the levels along with the associated dates and times when a user's level was changed in one field. In addition, it stores all of the times as Unix timestamps. Here's an example of a typical field associated with a client:
a:20:{s:15:"1562695223.0001";s:6:"level0";s:15:"1562695223.0002";s:6:"level1";s:15:"1562695223.0003";s:6:"level2";s:15:"1562695223.0004";s:6:"level3";s:15:"1577906312.0001";s:11:"ccap_prepay";s:15:"1596575898.0001";s:12:"-ccap_prepay";s:15:"1596575898.0002";s:13:"ccap_graduate";s:15:"1596575898.0003";s:11:"ccap_prepay";s:15:"1596575898.0004";s:7:"-level3";s:15:"1597196952.0001";s:14:"-ccap_graduate";s:15:"1597196952.0002";s:12:"-ccap_prepay";s:15:"1597196952.0003";s:13:"ccap_graduate";s:15:"1597196952.0004";s:11:"ccap_prepay";s:15:"1598382433.0001";s:14:"-ccap_graduate";s:15:"1598382433.0002";s:12:"-ccap_prepay";s:15:"1598382433.0003";s:11:"ccap_prepay";s:15:"1598382433.0004";s:6:"level3";s:15:"1605290551.0001";s:12:"-ccap_prepay";s:15:"1605290551.0002";s:11:"ccap_prepay";s:15:"1605290551.0003";s:13:"ccap_graduate";}
There are four columns in this table: umeta_id; user_id; meta_key; meta_value. The data above is stored in the meta_value column.
You'll notice that it also has multiple ccap_* entries. CCAP stands for custom capapability and I would like to be able to chart those assignments and associated times as well.
Do you have any idea how I can accomplish this?
Thank you for any help you can give.
I talked to an engineer about this and he told me that I would need to learn Python and I believe he said I would need to learn how to also use Pandas and Numpy to extract the data I need but he wasn't exactly sure. I started taking a data analyst course on Coursera but I still haven't learned what I need to learn and it's already been several months. It would be great if someone could provide a solution that I could implement more quickly and use on an ongoing basis.
If there's a way to accomplish my goal by exporting this table to a CSV file and using Microsoft Excel or Google Sheets, I'm open to that too.
Here's an image of the table (if it helps):
Database table
Here's an example of my desired output:
Desired output
In my desired output, I used Excel and created a column that converts the Unix timestamp to a short date and another column where I used a nested IF statement to convert the CCAP or level to its meaning that we understand internally.
I am building a web application that will run off of data that is produced for the public by a governmental agency. The issue is that the csv file that houses the data I need is a 2,000 column beast of a file. The file is what it is, I need to find the best way to take it and modify it. I know I need to break this data up into much smaller tables within MySQL, but I'm struggling with the best way to do this. I need to make this as easy as possible to replicate for next year when the data file is produced again (and every year after). I've searched for programs to help, and everything I've seen deals with a huge amount of rows, not columns. Has anyone else dealt with this problem before? Any ideas? I've spent the last week color coding columns in excel and moving data to new tabs, but this is time consuming, will be super difficult to replicate and I worry it leaves me open for copy and paste errors. I'm at a complete loss here!
Thank you in advance!
I suggest that you use functions in excel to give every column an automatic name "column1", "column2", "column3", etc.
After that import the entire csv file into MySQL.
Decide on which columns you want to group together into separate tables. This is the longest step and no program can help you manage this part.
Query your massive SQL table to get just the columns you want for each group. Export these queries to CSV and then import them as new tables in your database.
At the end, if you want, query all the columns you didn't put into separate groups. Make this a new table in the database and delete the original table to save on storage space.
Does this government csv file get updated and republished in the same format every time? If so you'll need to write a script to do all of the above automatically.
So I'm sure all of you know what a wage report is, it's basically a piece of paper where you write how many hours you worked that day and how much you are making per hour and then you sign a date. So I have an excel version of a physical wage report and I want to be able to take the data from an SQL server and insert it certain pieces of data in certain cells in the excel sheet, because it needs to be put in a certain format. I already know how you can just insert the data from the SQL server into a table in an excel file, but my question is how do I do it if I need the info from SQL displayed in a certain way on the excel sheet.
I haven't tried anything, because I've been searching for a way to do this, but I've come up with nothing. I'm having a hard time knowing where to start, if this is even possible at all.
We are trying to get the CSV importer to work properly and we're having issues because it's creating duplicate products. Ultimately we want to import hundreds of product variations, but we're not close to getting that working yet. We've eliminated all of the possible variables that we can think of to getting this to work and we have a pretty simple test that's failing.
What we did was this:
Export all of the products (WooCommerce >> CSV Import Suite >>
Export Products Tab, Limit = unlimited, Offset=0, Columns = All
Columns. We've tested it with "include hidden data" checked and
unchecked.)
Save the CSV file to the desktop (Windows) and didn't
open it or edit it in any way.
Click on the import button, upload the files and click on the final button to start the process.
I would expect it to skip every product in the import file because it already exists in the database, but it routinely adds 8 of the 67 products as new ones. Each time we've tested has been the same 8 products and the option for including hidden data on the export doesn't impact the results.
Has anyone seen this issue? Any ideas on a workaround or fix?
If not, does anyone have any suggestions on how to de-duplicate the records?
Check the ID column. If it's a new product you're gonna want to make the ID cell blank.
woocommerce CSV product import suite does not need the post id but to know whether you are updating creating or removing the correct attributes; from a variation you will always need a unique sku , if you just use the parents sku or the same for every variation you end up with duplicates or missing variation; due to attributes changing but sku being the same.
Woocommerce seems to care more for the sku over attributes it is also not smart enough to account for missing attributes. GIST is UNIQUE SKU ALWAYS, if uploading to existing products and not sure if attributes have changed; use override to remove blank cells this will clear unwanted attributes not in use.
am about to make an addition to my database that might make life much easier for users that are not normally accustomed to using it.
I have a table in my database; which holds Historical_stock_data and Historical_currency_data (Values to convert Share Price into dollars) in the other table. Each of them have basically got four & three columns (StockCode, Dates, SharePrice, Volume) and (CurrencyCode, Dates, Rates) respectively.
Now at the end of each trading day; I need to update the both tables. There around 200 companies in the database; and currently I basically have a portfolio in Yahoo finance holding all 200 companies which I download into Excel and edit so it matches the columns; and then manually update the database by adding to the table. (I can use the transfer spreadsheet method as well to do this but I ran into a problem once with it so do it manually now; my coding skills are not the greatest)
The currency prices, since there are only around 6 currencies in the database to dollars; I update them manually; this is not an issue at the moment; but if more companies from different countries are included than it would be easier; and besides it is always better to automate things; so was wondering if this was possible as well.
So if there was a way to update these values directly into Access at the end of each day without having to bother with Excel; that would be great. If there is then if I'm shown an example of what sort of code to use with just one example; I should be able to finish it.
Search on "yahoo stock price excel" and you should find lots of sample code. Then convert the code to using recordsets/tables instead of cells and you're done.