Does anyone know how I can use RMySQL (or another library) to update a row in a table, rather than having to pull out the full table and push it back in? I don't want to read such a huge table into memory just to update one row.
What I am trying to do is pull out a row, change some of the values in there within R and push the same row object back into the table.
However, dbWriteTable seems to replace the entire table rather than just the row I specify.
The easiest way is to construct a string within R containing the adequate SQL Update statement and use dbSendQuery to push your data back into the table.
Using sqldf package:
library(sqldf)
table_name = data.frame(a = 1:10, b = 4)
# Open connection
sqldf()
fn$sqldf("update table_name set b=1")
ans = sqldf("select * from main.table_name")
# Close connection
sqldf()
print(table_name)
Related
OK so I'm a complete Novice to SSIS but i needed to export images stored in our DB relating to specific sales orders , I attempted to do this in as an SP in SQ but this required CURSORS and I found it very easy to do the same thing in SSIS but I have a bit of an odd question , The DataFlow OLE DB to Data Flow works fine but i have have had to Declare the path OUTPUT FILE LOCATION in the SQL . I have worked out how to create a Dynamic File creation On the control flow but what i cant work out is how to remove the declared PATH and point it to the Control flow FILE SYSTEM TASK . I really hope this make sense and I appriciate any assistance
Control Flow
Data Flow Task
SQL -
DECLARE #Path nvarchar(1000);
SET #Path = N'C:\Users\X-JonC\Documents\';
SELECT
Products_IMAGEDATA.ImageData
, Products_IMAGEDATA.ImageTitle
, #Path + Imagetitle AS path
FROM
SalesOrders
INNER JOIN
SalesOrderItems
ON SalesOrders.SalesOrder = SalesOrderItems.SalesOrder
INNER JOIN
Products
ON SalesOrderItems.Product = Products.Product
AND SalesOrderItems.Product = Products.Product
INNER JOIN
Products_IMAGEDATA
ON Products.Product = Products_IMAGEDATA.Product
WHERE
SalesOrders.SalesOrderId = ?
A File System Task is something you use to perform operations on the file system (copy/rename/delete files/folders). You likely don't need a file system task unless you need to do something with the file after you've exported it to disk (like copy to a remote location or something).
A Data Flow Task is something you use to move data between multiple places. In your case, you'd like to move data from a database to a file system. The interesting twist is that you need to export binary/image data.
You have an excellent start by having created a data flow task and wired up an Export Column task to it.
The challenge that you're struggling with is how do I get my SSIS variable value into the package. Currently, you have it hard coded to a TSQL variable #Path. Assuming what you have is working, then you merely need to use the parameterization approach you already have with SalesOrderId ? and populate the value of #Path in the same manner- thus line three becomes SET #Path = ?;
One thing to note is that OLE and ODBC parameterization is based on ordinal position (0 and 1 based respectively). By adding in this new parameter placeholder in on line 3, it's now the first element as it comes before the WHERE clause's usage so you will need to update the mapping. Or you can be lazy and replace the empty line 2 with DECLARE #SalesOrderId int = ?; {this allows the first element to remain as is, you add your new element as +1 to the usage}. You'd need to replace the final question mark with the local variable like
DECLARE #Path nvarchar(1000);
DECLARE #SalesOrderId = ?
SET #Path = ?;
SELECT
Products_IMAGEDATA.ImageData
, Products_IMAGEDATA.ImageTitle
, #Path + Imagetitle AS path
FROM
SalesOrders
INNER JOIN
SalesOrderItems
ON SalesOrders.SalesOrder = SalesOrderItems.SalesOrder
INNER JOIN
Products
ON SalesOrderItems.Product = Products.Product
AND SalesOrderItems.Product = Products.Product
INNER JOIN
Products_IMAGEDATA
ON Products.Product = Products_IMAGEDATA.Product
WHERE
SalesOrders.SalesOrderId = #SalesOrderId;
Reference answers
Using SSIS to extract a XML representation of table data to a file
Export Varbinary(max) column with ssis
I have the following Sequence Container inside of a ForEach loop in my SSIS package:
I am busy testing the ROLLBACK TRANSACTION statement, it executes fine but it does not rollback.
Not sure if I am missing anything?
Thanks in advance.
EDIT:
This is how the data flow looks like in my Sequence Container:
I took an alternative route to make sure that my fail over works. What I did was I added a column to my tables that will store a GUID and when there is an error I just delete records from tables where that GUID is equal to the one in that is used in that session. I added the GUID in my select statement that my ForEach Container will use and then I write it to the table.
SELECT
ReconMedicalAidFile.fReconMedicalAidFileID
,ReconMedicalAidFile.fReconMedicalAidID
,ReconMedicalAids.fMedicalAidID
,ReconMedicalAidFile.fFileName
,ReconMedicalAidFile.fFileLocation
,ReconMedicalAidFile.fFileImportedDate
,ReconMedicalAidFile.fNumberRecords
,ReconMedicalAidFile.fUser
,ReconMedicalAidFile.fIsImported
,CONVERT(varchar(50),NEWID()) AS 'Session'
FROM ReconMedicalAidFile
INNER JOIN ReconMedicalAids ON ReconMedicalAidFile.fReconMedicalAidID = ReconMedicalAids.fReconMedicalAidID
WHERE (fIsImported = 0) AND (fNumberRecords = 0)
I added this code in the Script Task which will map the selected GUID above:
DELETE FROM BankmedStatments
WHERE fSession = ?
DELETE FROM ReconMedicalAidData
WHERE fSession = ?
I am using R to manage a MySQL database and ran into the following question: Can I use multiple cores to run an update on a table, or does SQL get confused by that? The R code I use works like this (pseudocode):
d_ply(.data = my_local_table, .variables = ~ID, .fun = function(x){
check if ID exists in my_DB_table
if(exists):
Update record
else:
Create new record
Is it problematic to use the .parallel option of the plyr package in this case?
What I'm trying to accomplish.
My site is live, and I'm making some changes to it.
When I'm ready to go live with the changes, I will essentially drop the old database (myFirstDb) and put the new one up(mySecondDb).
BUT - I will want to keep a few of the tables from the live site(myFirstDb) - and bring them into the new database (mySecondDb).
I know I can do this via phpMyAdmin - but that only allows one table at a time to be copied.
MY QUESTION:
What kind of SQL Query in mySQL will allow me to define the tables to move from database one to database two. (Keeping the structure and data etc)
So here's what I have so far.
First off, is it possible to create a loop in mySql - SQL query? If so...
Could someone assist me proper syntax for creating an array in mySQL?
How would I go about creating the array in valid sql syntax?
// define array of table names here. <-- how to form this?
SET #table_name_array = ('tableFive', 'tableSeven', 'tableNine', 'tableFifteen', 'tableNineteen', 'tableNth', 'tableMaybeOneMore');
How would I go about looping thru that array in proper sql syntax?
// start loop thru each table name <-- how to write this?
My code so far
SET #database_name := 'myFirstDb';
SET #second_database_name := 'mySecondDb';
// define array of table names here. <-- how to form this?
SET #table_name_array = ('tableFive', 'tableSeven', 'tableNine', 'tableFifteen', 'tableNineteen', 'tableNth', 'tableMaybeOneMore');
// start loop thru each table name <-- how to write this?
SET #table_name := 'value';
SET #second_db_table_name := #table_name;
CREATE TABLE #second_db_table_name LIKE #table_name
ALTER TABLE #second_db_table_name DISABLE KEYS
INSERT INTO #second_db_table_name SELECT * FROM #table_name
ALTER TABLE #table_name ENABLE KEYS
//end loop here
This is my first attempt to throw data back and forth between a local MySQL database and R. That said, I have a table created in the database and want to insert data into it. Currently, it is a blank table (created with MySQL Query Browser) and has a PK set.
I am using the RODBC package (RMySQL gives me errors) and prefer to stick with this library.
How should I go about inserting the data from a data frame into this table? Is there a quick solution or do I need to:
Create a new temp table from my dataframe
Insert the data
Drop the temp table
With separate commands? Any help much appreciated!
See help(sqlSave) in the package documentation; the example shows
channel <- odbcConnect("test")
sqlSave(channel, USArrests, rownames = "state", addPK=TRUE)
sqlFetch(channel, "USArrests", rownames = "state") # get the lot
foo <- cbind(state=row.names(USArrests), USArrests)[1:3, c(1,3)]
foo[1,2] <- 222
sqlUpdate(channel, foo, "USArrests")
sqlFetch(channel, "USArrests", rownames = "state", max = 5)
sqlDrop(channel, "USArrests")
close(channel)
which hopefully should be enough to get you going.