In magento I am importing bulk products. Every month I gets CSV of whole data(all products). I want to upload only those product which attributes value actually got changed.
For example :
If I have 5 products in magento. I have csv with those 5 products. From those 5 products only 1 product's description get changed in new csv. So I want to import only that changed product.
If this is not possible then can we get all changed products after import ?
Thanks.
Usually, you should get an incremental csv, with only the data that's changed.
One thing that you could potentially do is, after loading the product and setting new data from the CSV, you can use $product->dataHasChangedFor($field) to determine if the new data is different than the original one for the particular field.
To see more about how this works, you can check the implementation in Varien_Object. Basically Magento stores original data that's loaded separately, so it allows comparing it with the newly set data.
Cheers.
Related
On our Wordpress site, we use a plugin called s2member and it stores the levels (roles) of our clients as well as the times they were assigned a specific level in our database. I would like to create a table that shows when a user was assigned a specific level. I'm having a challenge getting the data I need because of the way the data is stored in the field. It stores all of the levels along with the associated dates and times when a user's level was changed in one field. In addition, it stores all of the times as Unix timestamps. Here's an example of a typical field associated with a client:
a:20:{s:15:"1562695223.0001";s:6:"level0";s:15:"1562695223.0002";s:6:"level1";s:15:"1562695223.0003";s:6:"level2";s:15:"1562695223.0004";s:6:"level3";s:15:"1577906312.0001";s:11:"ccap_prepay";s:15:"1596575898.0001";s:12:"-ccap_prepay";s:15:"1596575898.0002";s:13:"ccap_graduate";s:15:"1596575898.0003";s:11:"ccap_prepay";s:15:"1596575898.0004";s:7:"-level3";s:15:"1597196952.0001";s:14:"-ccap_graduate";s:15:"1597196952.0002";s:12:"-ccap_prepay";s:15:"1597196952.0003";s:13:"ccap_graduate";s:15:"1597196952.0004";s:11:"ccap_prepay";s:15:"1598382433.0001";s:14:"-ccap_graduate";s:15:"1598382433.0002";s:12:"-ccap_prepay";s:15:"1598382433.0003";s:11:"ccap_prepay";s:15:"1598382433.0004";s:6:"level3";s:15:"1605290551.0001";s:12:"-ccap_prepay";s:15:"1605290551.0002";s:11:"ccap_prepay";s:15:"1605290551.0003";s:13:"ccap_graduate";}
There are four columns in this table: umeta_id; user_id; meta_key; meta_value. The data above is stored in the meta_value column.
You'll notice that it also has multiple ccap_* entries. CCAP stands for custom capapability and I would like to be able to chart those assignments and associated times as well.
Do you have any idea how I can accomplish this?
Thank you for any help you can give.
I talked to an engineer about this and he told me that I would need to learn Python and I believe he said I would need to learn how to also use Pandas and Numpy to extract the data I need but he wasn't exactly sure. I started taking a data analyst course on Coursera but I still haven't learned what I need to learn and it's already been several months. It would be great if someone could provide a solution that I could implement more quickly and use on an ongoing basis.
If there's a way to accomplish my goal by exporting this table to a CSV file and using Microsoft Excel or Google Sheets, I'm open to that too.
Here's an image of the table (if it helps):
Database table
Here's an example of my desired output:
Desired output
In my desired output, I used Excel and created a column that converts the Unix timestamp to a short date and another column where I used a nested IF statement to convert the CCAP or level to its meaning that we understand internally.
I am building a web application that will run off of data that is produced for the public by a governmental agency. The issue is that the csv file that houses the data I need is a 2,000 column beast of a file. The file is what it is, I need to find the best way to take it and modify it. I know I need to break this data up into much smaller tables within MySQL, but I'm struggling with the best way to do this. I need to make this as easy as possible to replicate for next year when the data file is produced again (and every year after). I've searched for programs to help, and everything I've seen deals with a huge amount of rows, not columns. Has anyone else dealt with this problem before? Any ideas? I've spent the last week color coding columns in excel and moving data to new tabs, but this is time consuming, will be super difficult to replicate and I worry it leaves me open for copy and paste errors. I'm at a complete loss here!
Thank you in advance!
I suggest that you use functions in excel to give every column an automatic name "column1", "column2", "column3", etc.
After that import the entire csv file into MySQL.
Decide on which columns you want to group together into separate tables. This is the longest step and no program can help you manage this part.
Query your massive SQL table to get just the columns you want for each group. Export these queries to CSV and then import them as new tables in your database.
At the end, if you want, query all the columns you didn't put into separate groups. Make this a new table in the database and delete the original table to save on storage space.
Does this government csv file get updated and republished in the same format every time? If so you'll need to write a script to do all of the above automatically.
We are trying to get the CSV importer to work properly and we're having issues because it's creating duplicate products. Ultimately we want to import hundreds of product variations, but we're not close to getting that working yet. We've eliminated all of the possible variables that we can think of to getting this to work and we have a pretty simple test that's failing.
What we did was this:
Export all of the products (WooCommerce >> CSV Import Suite >>
Export Products Tab, Limit = unlimited, Offset=0, Columns = All
Columns. We've tested it with "include hidden data" checked and
unchecked.)
Save the CSV file to the desktop (Windows) and didn't
open it or edit it in any way.
Click on the import button, upload the files and click on the final button to start the process.
I would expect it to skip every product in the import file because it already exists in the database, but it routinely adds 8 of the 67 products as new ones. Each time we've tested has been the same 8 products and the option for including hidden data on the export doesn't impact the results.
Has anyone seen this issue? Any ideas on a workaround or fix?
If not, does anyone have any suggestions on how to de-duplicate the records?
Check the ID column. If it's a new product you're gonna want to make the ID cell blank.
woocommerce CSV product import suite does not need the post id but to know whether you are updating creating or removing the correct attributes; from a variation you will always need a unique sku , if you just use the parents sku or the same for every variation you end up with duplicates or missing variation; due to attributes changing but sku being the same.
Woocommerce seems to care more for the sku over attributes it is also not smart enough to account for missing attributes. GIST is UNIQUE SKU ALWAYS, if uploading to existing products and not sure if attributes have changed; use override to remove blank cells this will clear unwanted attributes not in use.
I'm attempting to import a csv file in Microsoft Dynamics 2011. The file is an export of sales data from our website and I'm attempting to create new Account, Contact, Invoice, Order, Invoice Product and Order Product entities using this data. I'm using a multi-entity data map to do this.
When I attempt the import however, creating the invoice fails with the message "The lookup reference could not be resolved." in connection with the currency field. This field is a lookup tied to currency name which in all cases is "US Dollar". This is exactly the same name as the USD currency in our database, but for some reason the lookup refuses to assign it correctly. There are no other currencies in our database with this name.
Can anyone offer me an explanation as to why this is happening and any steps I can take to deal with it? Many thanks.
EDIT: More information about my problem.
It seems that problems occur when I try and perform a multiple entity import with duplicates. All the new entries I'm creating from this import is dependent on other new entries created with the same data. I'm creating accounts, then creating customers to assign to those accounts for example. If said accounts have duplicates already in the system, they are ignored. This is expected. However, when it comes to create the customer assigned to that account, the lookup fails with a "Duplicate lookup reference was found" error. This error occurs when there are more than one things in the database that fit the value I'm using with the lookup.
I'm not 100% sure, but I think it's counting both the account that was ignored and the original account that's already in the system when performing the lookup. Can anyone offer more insight on how I might work around this?
I don't know for sure if this is your issue but my guess it is.
In the import wizard you can select how he resolves your lookups.
i.e: when in your CSV file the column contains the name of the currency you will have to select the name field in your import wizard for the currency field.
In the screen here you can see then country is a lookup and the name of the country is the value of CSV file. It must match the field of the lookup. If I select "created on" It doesnt find the lookup. because no country exists where created on equals "Belgium". So you select the Name field of the lookup and that is how he will check your value in the CSV in the lookup field you defined.
1.Put records for each record type in a separate file.
2.Put records for different owners in a separate file.
3.Ensure that the first line of the file contains column headers and that they match the CRM attribute display names EXACTLY if you would like the import wizard to automatically* map the data. *Automatic mapping is required if you intend to import records that are related to another record (See #6). I HIGHLY RECOMMEND USING AUTOMATIC MAPPING EVEN IF YOU DO NOT DESIRE RELATED RECORDS.
4.Make sure the first column heading is not the name of a record type.
5.Make sure data exists in your source file for all CRM required fields (denoted with a red asterisk on the CRM form).
6.Make sure some data exists for each column. If a column has no data delete the column.
7.If your data should be related to another record (i.e. Adding Contacts to existing Accounts) make sure the column header matched the CRM lookup attribute display name. If you have two different attributes in CRM with the same display name you will receive an error (EX. if you created a custom State picklist value to replace the text version and keep the display name the same "Address 1: State/Province"). Also make sure the company name values will match a corresponding account record (i.e. contacts with a Parent Customer value of Microsoft Corp. will not be added to the CRM Account Microsoft) and that there are no duplicate Accounts in the system as the import will not be able to resolve the proper Parent Customer.
8.If you are importing into a drop-down list, ensure all the values exist within CRM and there are no typographic errors and consistent capitalization in the source file.
EDIT: It seems like indeed there are multiple records in your DB that can fit in that 1 lookup. it's normal that he gives an error on this as he can't resolve the lookup. The CRM doesn't know which record to take.
Also you might want to watch out if you are importing a M-N Relationship. You will have to do a M-N Relationship with code. The standard import wizard cannot handle Many-to-many relationships
I have flat file that structured in a hierarchical format that looks something like this:
Area|AreaCode|AreaDescription
Region|RegionCode|RegionDescriptoin
Zone|ZoneCode|ZoneDescription
District|DistrictCode|DistrictDescription
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
RouteFooter
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
RouteFooter
DistrictFooter
District|DistrictCode|DistrictDescription
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
Record|Name|Address|Ect
RouteFooter
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
RouteFooter
DistrictFooter
ZoneFooter
RegionFooter
AreaFooter
I have to bring this into SSIS and consume information about the Record row and also about the header for the current record row. As well as information from several other sources and output a more simple flat file as a result.
I would like to read the flat file above into a structure that each row contains a record with the appropriate header information included.
My question is, what is the best way to do this if it is even possible?
First how do you tell what type of line you are on if you are on say line 3,987,986? How do you tell what is related to what? Is there apossiblity you could get this in a better format? Before spending lots of time (and don't kid yourself, this will take lots of time to set up and test properly) I would kick the file back to the provider and request it in a different format. You won't always get it, but you should at least try.
When I have done this in the past in DTS, the first characters of each line told me which structure the line referred to. I imported all into a staging table with two columns, one for the recordtype data and one for the rest. Then I parsed the rest into the staging tables for the type of record with the correct column structure for that type of record (and any fileds you might need to do the relationships) and then did clean up and then imported to prod tables. AS you also have differnt number of columns I would try that approach (only you may have to manually populate some columns instead of figuring out directly from the file), also give each record an identity filed in the staging tables. this will help you figure out the realtionships I think.