I am fairly new to programming and I've been trying to implement this code in Python, using PyCharm. I'm running the code via a remote server, using PyCharm on my local computer. It was written by a former colleague, and has been giving a lot of encoding issues since we updated the packages like MySQL and the Python interpreter to 3.8. The MySQL version is 8.0, but this is an update. That was not the version installed originally when the code was written.
This is the full error that I am getting:
findBestMatch
*** WARNING: FoundCity[i] = {'Country': 'Austria', 'Page': 'Contact', 'Confidence': 10, 'Mentions': 1}
WriteToDB
Problem <class 'MySQLdb._exceptions.ProgrammingError'>
Traceback (most recent call last):
File "/remotepath/TextMining/NER/FindLocationStoreSQL.py", line 399, in
WriteToDB(c.title(), cn, idProject, 10, "Contact", "v1", cursor, db, database_country)
File "/remotepath/TextMining/NER/FindLocationStoreSQL.py", line 286, in WriteToDB
cursor.execute(sql,values)
File "/home/localhost/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/home/localhost/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/home/localhost/.local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'City text mined, Country from datasource'',Confidence='10',FoundWhere=''Contact'' at line 1")
Process finished with exit code -1
The code sample of the function it is trying to run is below:
def findBestMatch(FoundCity,FoundCountry,database_country):
pair_candidates = []
for i in range(0,5):
for j in range(0,5):
if len(FoundCity)>i and len(FoundCountry)>j:
city_i = FoundCity[i].get('City')
country_j = FoundCountry[j].get('Country')
if city_i != None and country_j != None:
#sql = "SELECT City,Country_CountryName,Longitude,Latitude FROM Semanticon.City where city like '{0}' and Country_CountryName like '{1}' and Population>0 order by Population desc".format(FoundCity[i]['City'].encode('utf-8'),FoundCountry[j]['Country'].encode('utf-8'))
sql = "SELECT City,Country_CountryName,Longitude,Latitude FROM Semanticon.City where city like '{0}' and Country_CountryName like '{1}' and Population>0 order by Population desc".format(
FoundCity[i]['City'], FoundCountry[j]['Country'])
try:
cursor.execute(sql)
except:
db = MySQLdb.connect(host, username, password, database, charset='utf8')
db.set_character_set("utf8")
cursor = db.cursor()
cursor.execute(sql)
resul = cursor.fetchall()
if len(resul)>0:
pair_candidates.append({"City":FoundCity[i]['City'],"Country":FoundCountry[j]['Country'],"Score":(FoundCity[i]["Confidence"]+FoundCountry[j]["Confidence"]+0.5*(FoundCity[i]["Mentions"]+FoundCountry[j]["Mentions"]))})
#return FoundCity[i]['City'],FoundCountry[j]['Country'],FoundCity[i]['Confidence']
else:
if city_i == None:
print("*** WARNING: FoundCity[i] = ", FoundCity[i])
else:
print("*** WARNING: FoundCountry[j] = ", FoundCountry[j])
I had to take the encoding out, hence the commented out 'sql' line. The encoding was causing problems and adding an extra 'b' to the string to be read from the database.
The 'WriteToDB' function that it's complaining about is below:
def WriteToDB(City,Country,ProjectId,Confidence,Location,Version,cursor,db,database_country):
sql = None
if database_country!="":
if database_country == Country:
if City != "":
#sql = "UPDATE ProjectLocation SET City='{0}',DataTrace='{1}',Confidence={2},FoundWhere='{3}' WHERE Projects_idProjects={4} and Country='{5}';".format(City," City text mined, Country from datasource",Confidence,Location,ProjectId,original_database_cntry)
sql = "UPDATE ProjectLocation SET City='%s',DataTrace='%s',Confidence='%s',FoundWhere='%s' WHERE Projects_idProjects='%s' and Country='%s';"
values = (City, " City text mined, Country from datasource", Confidence, Location, ProjectId,
original_database_cntry)
if database_country!=Country:
if Country.encode('utf-8') in database_country:
#sql = "UPDATE ProjectLocation SET City='{0}',DataTrace='{1}',Confidence={2},FoundWhere='{3}' WHERE Projects_idProjects={4} and Country='{5}';".format(
# City, " City text mined, Country from datasource", Confidence, Location, ProjectId,
#original_database_cntry)
sql = "UPDATE ProjectLocation SET City='%s',DataTrace='%s',Confidence='%s',FoundWhere='%s' WHERE Projects_idProjects='%s' and Country='%s';"
values = (City, " City text mined, Country from datasource", Confidence, Location, ProjectId,
original_database_cntry)
else:
print("Country conflict in project:"+str(ProjectId))
else:
#sql = "Insert into ProjectLocation (Type,City,Country,Projects_idProjects,Original_idProjects,IsLocationFromDataset,Confidence,FoundWhere,Version,DataTrace)" \
# "Values ('{0}','{1}','{2}',{3},{4},0,{5},'{6}','{7}','{8}')".format("Main",City,Country,ProjectId,ProjectId,Confidence,Location,Version,"Both minded from text v0.1")
sql = "Insert into ProjectLocation (Type,City,Country,Projects_idProjects,Original_idProjects,IsLocationFromDataset,Confidence,FoundWhere,Version,DataTrace)" \
"Values ('%s','%s','%s','%s',0,'%s','%s','%s','%s','%s')"
values = ("Main", City, Country, ProjectId,ProjectId, Confidence, Location, Version, "Both minded from text v0.1")
if sql!=None:
cursor.execute(sql,values)
db.commit()
I commented out the SQL queries as shown and tried to bind them instead, because it was giving a lot of encoding errors. I'm not sure how to get rid of this error and not end up with the encoding errors yet again.
Can someone help?
UPDATE to the question.
I reverted back all the sql queries, and used the queries with the encoding now (previously commented out) and I am getting 'b's in the output.
Any suggestions on how to properly encode these SQL queries so the binary encoding does not come out as b's in the output?
Here is a sample of the output:
ProjectID,ProjectName,FoundCity,FoundCountry,DatabaseCity,DatabaseCountry,Confidence,FoundWhere,Website
2542, Migrantour Country,,b'',b'',b'',10,Contact,link
3938,GeoSmartCity,,,b'',b'',10,Contact,link
I am using project deployment. I have several project parameters. My packages only use project-level parameters, and no package-level ones. I have programatically deployed my project and set an environmental reference:
I call each package from a SQL Agent job. I am unable to link my environment variables to the package when it runs. I have successfully linked the project to the environment:
But now when I run my agent job, it fails. When I look at the SSISDB reports, it says it "created execution", but shows no variables.
Do I actually have to explicitly link every variable in each package to the environment variable? Why even bother to group them by environment?
I have created my environmental references like this (sql cmd):
EXEC [SSISDB].[catalog].[create_environment_reference] #environment_name='$(ChooseEnvironment)', #reference_id=#reference_id OUTPUT, #project_name='$(ProjectName)', #folder_name='$(folderName)', #reference_type=R
EXEC SSISDB.catalog.set_object_parameter_value #parameter_name=N'EmailFrom', #parameter_value='EmailFrom', #project_name=$(ProjectName), #object_type=20, #folder_name=$(FolderName), #value_type=N'R'
Additional info: I have created a sql agent job that calls each package with a job step like this:
set #cmd = N'/ISSERVER "\"\SSISDB\CHAT\SSISPackages\Chat_Load_RMS_InputFiles.dtsx\"" /SERVER "\"' + #TargetDBServer + '\"" /Par "\"$ServerOption::LOGGING_LEVEL(Int16)\"";1 /Par "\"$ServerOption::SYNCHRONIZED(Boolean)\"";True /CALLERINFO SQLAGENT /REPORTING E'
EXEC #ReturnCode = msdb.dbo.sp_add_jobstep #job_id=#jobId,
#step_name=N'PACKAGE: Chat_Load_RMS_InputFiles.dtsx',
#step_id=1,
#cmdexec_success_code=0,
#on_success_action=3,
#on_success_step_id=0,
#on_fail_action=2,
#on_fail_step_id=0,
#retry_attempts=0,
#retry_interval=0,
#os_run_priority=0, #subsystem=N'SSIS',
#command=#cmd,
#database_name=N'master',
#flags=0
Do I need to add a reference id to my SSIS #cmd variable? Also, if I address this in the job, can I remove my code above to set each project-level variable to an environment, or do I still need that? It seems like for cleanliness, I should just be able to say: this project uses this environment. Done. Otherwise, it's almost like using package-level variables and all the tinkering those require.
If you are running a package as an "direct" SSIS step in SQL Agent step, you have to select the environment in the package configuration tab on the step configuration dialog.
If you running it using TSQL script you need to provide a reference id when calling catalog.create_execution:
DECLARE
#reference_id bigint,
#FullPackageName NVARCHAR(100);
SELECT #reference_id = reference_id
FROM [$(SSISDB)].catalog.environment_references er
INNER JOIN [$(SSISDB)].catalog.projects AS p
ON p.project_id = er.project_id
INNER JOIN [$(SSISDB)].catalog.folders AS f
ON f.folder_id = p.folder_id
WHERE er.environment_folder_name IS NULL
AND er.environment_name = #EnvironmentName
AND p.name = #ProjectName
AND f.name = #FolderName;
IF ##ROWCOUNT = 0
BEGIN
DECLARE
#msg NVARCHAR(100);
SET #msg = N'Could not find a reference for a local (.) ''' + #EnvironmentName + N''' environment.';
THROW 50000, #msg, 1;
END;
SET #FullPackageName = #PackageName + N'.dtsx';
EXEC [$(SSISDB)].catalog.create_execution
#package_name = #FullPackageName,
#execution_id = #execution_id OUTPUT,
#folder_name = #FolderName,
#project_name = #ProjectName,
#use32bitruntime = False,
#reference_id = #reference_id;
I am trying to connect to a mysql server using LuaSql via a mysql proxy. I try to execute a simple program (db.lua):
require("luasql.mysql")
local _sqlEnv = assert(luasql.mysql())
local _con = nil
function read_auth(auth)
local host, port = string.match(proxy.backends[1].address, "(.*):(.*)")
_con = assert(_sqlEnv:connect( "db_name", "username", "password", "hostname", "3306"))
end
function disconnect_client()
assert(_con:close())
end
function read_query(packet)
local cur = con:execute("select * from t1")
myTable = {}
row = cur:fetch(myTable, "a")
print(myTable.id,myTable.user)
end
This code executes well when I execute it without mysql-proxy. When I am connecting with mysql-proxy, the error-log displays these errors:
mysql.lua:8: bad argument #1 to 'insert' (table expected, got nil)
db.lua:1: loop or previous error loading module 'luasql.mysql'
mysql.lua is a default file of LuaSql:
---------------------------------------------------------------------
-- MySQL specific tests and configurations.
-- $Id: mysql.lua,v 1.4 2006/01/25 20:28:30 tomas Exp $
---------------------------------------------------------------------
QUERYING_STRING_TYPE_NAME = "binary(65535)"
table.insert (CUR_METHODS, "numrows")
table.insert (EXTENSIONS, numrows)
---------------------------------------------------------------------
-- Build SQL command to create the test table.
---------------------------------------------------------------------
local _define_table = define_table
function define_table (n)
return _define_table(n) .. " TYPE = InnoDB;"
end
---------------------------------------------------------------------
-- MySQL versions 4.0.x do not implement rollback.
---------------------------------------------------------------------
local _rollback = rollback
function rollback ()
if luasql._MYSQLVERSION and string.sub(luasql._MYSQLVERSION, 1, 3) == "4.0" then
io.write("skipping rollback test (mysql version 4.0.x)")
return
else
_rollback ()
end
end
As stated in my previous comment, the error indicates that table.insert (CUR_METHODS, ...) is getting a nil as first arg. Since the first arg is CUR_METHODS, it means that this object CUR_METHODS has not been defined yet. Since this happens near top of the luasql.mysql module, my guess is that the luasql initialization was incomplete, maybe because the mysql DLL was not found. My guess is that the LUA_CPATH does not find the MySQL DLL for luasql, but I'm surprised that you wouldn't get a package error, so something odd is going on. You'll have to dig into the luasql module and C file to figure out why it is not being created.
Update: alternately, update your post to show the output of print("LUA path:", package.path) and print("LUA path:", package.cpath) from your mysql-proxy script and also show the path of folder where luasql is installed and contents of that folder.
I am not able to execute the procedure using one shared folder path in sql server 2008.
Below is the piece of the stored procedure i am using:
DECLARE #S AS NVARCHAR(36) = NEWID(), #return_value int,
> #vcStatusMsg nvarchar(max), #vcFileLocation nvarchar(500)
> EXEC #return_value = [dbo].[uspGenerateReport]
> #vcStatusMsg = #vcStatusMsg OUTPUT,
> #vcFileLocation = #vcFileLocation OUTPUT,
> #vcReportPath = N'\\demo\',
> #vcReportFileName = #S,
> #vcReportName = N'UserReport',
> #vcReportFolder = N'testReport', #vcReportParamter = N'TEST_ID=10', #vcReportServerUrl =
> N'http://***/reportserver/reportexecution2005.asmx?wsdl',
> #vcServerDomain = N'***', #vcServerUserName = N'***',
> #vcServerPassword = N'***'
>
> SELECT #vcStatusMsg as N'#vcStatusMsg', #vcFileLocation as
> N'#vcFileLocation'
On executing the above procedure, i am getting the error message :
"System.IO.IOException: Logon failure: unknown user name or bad password"
Here i am trying to access the shared folder path.
Thanks in advance.
This error message is due to the invalid path in the procedure.
The network path specified in the procedure requires authentication and sql server didn't get the valid credential to access the file so its creating the error message.
So the solution would be something as below,
Specify the credentials in procedure to access the shared folder.
or
If you are unable/ don't want to set the credential, just remove the password protection from the shared folder.
Hope this might help you.