Django 1.11.7
MySQL
I was trying to change the value of an object like this:
# change the value of the filed and save
def patch(...):
instance.field_name = new_name
instance.save()
print(instance.filed_name)
When I run the code I got the print result as new_name. But when I check the database manually I got the result as old_name.
Then I tried ways like:
instance.save(update_fields=['field'])
and
ModelName.objects.filter(id=instance.id).update(field_name=new_name)
but get the above problem as well. And meanwhile, the project runs perfectly functional except for this segment of code.
Any idea what caused this problem or suggestion on how to solve it?
Is that piece of code inside a transaction? Maybe the transaction gets rolledback somewhere later.
When you read from the DB are you inside a transaction? Some transaction modes may not show you this change.
Are you sure that field_name is the correct field name? Maybe you have a typo and you just set a property of the object without changing model field. From what I see you sometimes type "field_name" and sometimes "filed_name"
Problem.
I regularly receive a feed files from different suppliers. Although the column names are consistent the problem comes when some suppliers send text files with more or less columns in there feed file.
Furthermore the arrangement of these files are inconsistent.
Other than the Dynamic data flow task provided by Cozy Roc is there another way I could import these files. I am not a C# guru but i am driven torwards using a "Script Task" control flow or "Script Component" Data flow task.
Any suggestion, samples or direction will greatly be appreciated.
http://www.cozyroc.com/ssis/data-flow-task
Some forums
http://www.sqlservercentral.com/Forums/Topic525799-148-1.aspx#bm526400
http://www.bidn.com/forums/microsoft-business-intelligence/integration-services/26/dynamic-data-flow
Off the top of my head, I have a 50% solution for you.
The problem
SSIS really cares about meta data so variations in it tend to result in exceptions. DTS was far more forgiving in this sense. That strong need for consistent meta data makes use of the Flat File Source troublesome.
Query based solution
If the problem is the component, let's not use it. What I like about this approach is that conceptually, it's the same as querying a table-the order of columns does not matter nor does the presence of extra columns matter.
Variables
I created 3 variables, all of type string: CurrentFileName, InputFolder and Query.
InputFolder is hard wired to the source folder. In my example, it's C:\ssisdata\Kipreal
CurrentFileName is the name of a file. During design time, it was input5columns.csv but that will change at run time.
Query is an expression "SELECT col1, col2, col3, col4, col5 FROM " + #[User::CurrentFilename]
Connection manager
Set up a connection to the input file using the JET OLEDB driver. After creating it as described in the linked article, I renamed it to FileOLEDB and set an expression on the ConnectionManager of "Data Source=" + #[User::InputFolder] + ";Provider=Microsoft.Jet.OLEDB.4.0;Extended Properties=\"text;HDR=Yes;FMT=CSVDelimited;\";"
Control Flow
My Control Flow looks like a Data flow task nested in a Foreach file enumerator
Foreach File Enumerator
My Foreach File enumerator is configured to operate on files. I put an expression on the Directory for #[User::InputFolder] Notice that at this point, if the value of that folder needs to change, it'll correctly be updated in both the Connection Manager and the file enumerator. In "Retrieve file name", instead of the default "Fully Qualified", choose "Name and Extension"
In the Variable Mappings tab, assign the value to our #[User::CurrentFileName] variable
At this point, each iteration of the loop will change the value of the #[User::Query to reflect the current file name.
Data Flow
This is actually the easiest piece. Use an OLE DB source and wire it as indicated.
Use the FileOLEDB connection manager and change the Data Access mode to "SQL Command from variable." Use the #[User::Query] variable in there, click OK and you're ready to work.
Sample data
I created two sample files input5columns.csv and input7columns.csv All of the columns of 5 are in 7 but 7 has them in a different order (col2 is ordinal position 2 and 6). I negated all the values in 7 to make it readily apparent which file is being operated on.
col1,col3,col2,col5,col4
1,3,2,5,4
1111,3333,2222,5555,4444
11,33,22,55,44
111,333,222,555,444
and
col1,col3,col7,col5,col4,col6,col2
-1111,-3333,-7777,-5555,-4444,-6666,-2222
-111,-333,-777,-555,-444,-666,-222
-1,-3,-7,-5,-4,-6,-2
-11,-33,-77,-55,-44,-666,-222
Running the package results in these two screen shots
What's missing
I don't know of a way to tell the query based approach that it's OK if a column doesn't exist. If there's a unique key, I suppose you could define your query to have only the columns that must be there and then perform lookups against the file to try and obtain the columns that ought to be there and not fail the lookup if the column doesn't exist. Pretty kludgey though.
Our solution. We use parent child packages. In the parent pacakge we take the individual client files and transform them to our standard format files then call the child package to process the standard import using the file we created. This only works if the client is consistent in what they send though, if they try to change their format from what they agreed to send us, we return the file.
Newbie question. I believe this is one of the most common errors. I found several of them on msdn forum itself but probably there are many ways on achieving this error? Please help.
I am trying to move and rename some images from one folder to another (and yes I have seen the blog by Rafael Salas and many others but none of them helps).
Like moving from \server1\images\123-456.jpg to \server2\images\123456.jpg
I am using a foreach.
Source variable is built dynamically. In the first iteration #imagePath = \server1\images\123-456.jpg ( i checked using messagebox.show)
I have defined #remoteImagePath = \server2\images\ (which never changes) and #revisedImageName = 123456.jpg (built dynamically in script task using string replace - also checked using messagebox.show)
In FileSystem Task, I am using SourceVariable as #imagePath and using Expressions to define Destination as in #[User::remoteImagePath] + "\" + #[User::revisedImageName]
Dont know for what reason, I am getting this error
Failed to lock variable "\server2\images\123456.jpg" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
Set the IsSourcePathVariable to True and set the SourceVariable to a SINGLE variable.
If you use the expression editor to set Source using multiple variables, or anything else apart from a SINGLE variable it will not work. If you want to concatenate hard-coded strings to variables, or multiple variables, do this by creating a new variable in the package, and use the Expressions tab in the variables tab to build the variable as a single variable.
I believe expressions editor needs syntax
#[User::remoteImagePath] + "\\" + #[User::revisedImageName]
For CSV file import, Silverstripe provides a good class CsvBulkLoader . But it requires the first row in the file to be column names. In my case, the customer files dont have header row, and the first row is also data.
Is there an easy way to do this in this case?
The problem is it wants to know which column will go to which field in the database.
You can set public $hasHeaderRow = false; in your custom CsvBulkUploader class. Example:
class MyCsvBulkUploader extends CsvBulkUploader {
public $hasHeaderRow = false;
//custom code here
}
Note that you'll need to write your own custom CsvBulkUploader for anything beyond the simplest case. There's some helpful tutorials on how to write a custom uploader at the bottom of the page at http://doc.silverstripe.org/framework/en/trunk/howto/csv-import
Actually after doing it, I found out that Silverstripe CsvBulkLoader class is not efficent. If file is big it runs out of memory. Also, it doesnt allow ploading zipfile but csv file is too large to upload.
I wrote my own custom class which is very fast and can read cvs file contents from zip.
In silverstripe to find duplicates, they search in list whether an object exists. It means each time it needs to check a record existance, it traverses the list from start. The csv file I have has near one million records. For silverstripe takes several ours to run it. But my class with checks duplicates by isset array key and also pulls existing records by raw sql. Silverstripe datamodel when looped makes another query for each loop which means near one million queries.
Raw sql runs very fast.
How do you check to see if a file does not exist in SQL Server Integration Services 2005?
Is there a native SSIS component which will just do this for you?
I have checked for file existence this using the Script Task and then branch accordingly.
You can do something like
If System.IO.File.Exists("\\Server\Share\Folder\File.Ext") Then
Dts.TaskResult = Dts.Results.Success
Else
Dts.TaskResult = Dts.Results.Failure
End If
Although there are no native components for this, there are several third party components for SSIS that you can use for this purpose.
The File System Task in SSIS is basically for move, copy, delete, etc., but does not support file existence checks.
#Raj More gave a good solution. Another way that I have used before is to create a Foreach Loop Container that loops over the file system for a file spec. If you know the name of the file you want, then you can set the name in a variable and set the spec to equal the variable in the expression tab for the Foreach Loop Container. You could also just specify or a directory or a partial file name if you don't know the exact name but know the naming convention or know there will be no other files in the folder.
If you want to take a specific action based on whether or not there is a file, then you could create a variable with a default value of 0 and create a script task in the Foreach Loop Container that increments the variable. You could also just put the commands in the Foreach Loop Container that you want to execute if you want to execute it for the existence of each individual file. If you want to take an action based on the absence of the file, then you could restrict your precedence constraint after the Foreach Loop Container so that it is restricted on constraint and expression and make it check if the counter variable is > 0.
#Raj's solution could also be used to increment the variable. Instead of using an If Else to raise an error or success result, you could do this:
C#
if (System.IO.File.Exists("\\Server\Share\Folder\File.Ext"))
{ Dts.Variables["my_case_sensitive_variable_name"].Value = Dts.Variables["my_case_sensitive_variable_name"].Value + 1;
}
VB.NET
If System.IO.File.Exists("\\Server\Share\Folder\File.Ext") Then
Dts.Variables["my_case_sensitive_variable_name"].Value = Dts.Variables["my_case_sensitive_variable_name"].Value + 1
End If
The advantage of this approach is that the package may not need to fail in the absence of a file. You could also use a variable name if the file changes that you could define either as a variable in the package or just solely created in the script task. The only short-coming of #Raj's approach is that you have to know the file name you want to check.
Another possibility is to execute a File System Task to rename the file to its existing name or copy the file to its existing location. If the file doesn't exist, then you can route the error to an action. I don't recommend this solution, but I remember using it years ago in one instance where it actually made sense. But in that particular instance, I was actually copying it to a real location.
Good luck!