Migrating MySQL blob (image) to FileMaker container using PowerShell - mysql

In searching I've found a number of other people that have tried, but none that have been successful.
Here's the problem. I want to take a bunch of images I have stored on my MySQL server in blobs and move them into FileMaker containers.
The best lead I've got is the putas command. It looks something like putas ('$Image','JPEG').
My particular application is as follows. $DataSet.Image1 is a JPEG file stored as "0xFFD8....". The data being in this format may well be the issue, but I don't know what I'd need to convert it to first.
$cmd.CommandText = "update Checklists set Image1 = PutAs('$($DataSet.Image1)', 'JPEG')"
$cmd.ExecuteNonQuery();
All I keep getting is syntax error, but I've tried the syntax many different ways I can't get it to go no matter what I do.
I'd very much like to see someone having success with this to post their example. Any other ideas or workarounds are welcome as well.
Edit:
Here is some extra info. Greg Lane at Skeleton Key gives this example, but I'm not sure how to translate it to PowerShell.
import java.sql.*; import java.io.*;
def url = "jdbc:filemaker://localhost/fmserver_sample";
def driver = "com.filemaker.jdbc.Driver";
def user = "admin";
def password = "";
System.setProperty("jdbc.drivers", driver);
connection = DriverManager.getConnection (url, user, password);
filename = "/Users/Greg/Pictures/vacation/DSC_0202.jpg";
file = new File (filename);
inputstream = new FileInputStream (filename);
sql = "INSERT INTO english_nature (ID, img) VALUES (-1, PutAs(?, 'JPEG'))";
pstatement = connection.prepareStatement ( sql );
pstatement.setBinaryStream (1, inputstream, (int)file.length ());
pstatement.execute ();
//cleanup
pstatement = null;
inputstream = null;
file = null;
connection.close();

I figured it out. For anyone in the future here is how you do it.
$cmd.CommandText = "update Checklists set Image1 = PutAs(?, 'JPEG') where serial = '$($DataSet.serial)' AND ChecklistNumber = 1"
$cmd.Parameters.Add('?', $DataSet.Image1)
$cmd.Prepare()
$cmd.ExecuteNonQuery();

Related

Inserting data into a SQL server from an excel file

First of all, sorry for my lack of knowledge regarding databases, this is my first time working with them.
I am having some issues trying to get the data from an excel file and putting it into a data base.
Using answers from the site, I managed to kind of connect to the database by doing this.
import pandas as pd
import pyodbc
server = 'XXXXX'
db = 'XXXXXdb'
# create Connection and Cursor objects
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=' + server + ';DATABASE=' + db + ';Trusted_Connection=yes')
cursor = conn.cursor()
# read data from excel
data = pd.read_excel('data.csv')
But I dont really know what to do now.
I have 3 tables, which are connected by a 'productID', my excel file mimics the data base, meaning that all the columns in the excel file have a place to go in the DB.
My plan was to read the excel file and make lists with each column, then insert into the DB each column value but I have no idea how to create a query that can do this.
Once I get the query I think the data insertion can be done like this:
query = "xxxxxxxxxxxxxx"
for row in data:
#The following is not the real code
productID = productID
name = name
url = url
values = (productID, name, url)
cursor.execute(query,values)
conn.commit()
conn.close
Database looks like this.
https://prnt.sc/n2d2fm
http://prntscr.com/n2d3sh
http://prntscr.com/n2d3yj
EDIT:
Tried doing something like this, but i'm getting 'not all arguments converted during string formatting' Type error.
import pymysql
import pandas as pd
connStr = pymysql.connect(host = 'xx.xxx.xx.xx', port = xxxx, user = 'xxxx', password = 'xxxxxxxxxxx')
df = pd.read_csv('GenericProducts.csv')
cursor = connStr.cursor()
query = "INSERT INTO [Productos]([ItemID],[Nombre])) values (?,?)"
for index,row in df.iterrows():
#cursor.execute("INSERT INTO dbo.Productos([ItemID],[Nombre])) values (?,?,?)", row['codigoEspecificoProducto'], row['nombreProducto'])
codigoEspecificoProducto = row['codigoEspecificoProducto']
nombreProducto = row['nombreProducto']
values = (codigoEspecificoProducto,nombreProducto)
cursor.execute(query,values)
connStr.commit()
cursor.close()
connStr.close()
I think my problem is in how I'm defining the query, surely thats not the right way
Try this, you seem to have changed the library from pyodbc to mysql, it seems to expect %s instead of ?
import pymysql
import pandas as pd
connStr = pymysql.connect(host = 'xx.xxx.xx.xx', port = xxxx, user = 'xxxx', password = 'xxxxxxxxxxx')
df = pd.read_csv('GenericProducts.csv')
cursor = connStr.cursor()
query = "INSERT INTO [Productos]([ItemID],[Nombre]) values (%s,%s)"
for index,row in df.iterrows():
#cursor.execute("INSERT INTO dbo.Productos([ItemID],[Nombre]) values (%s,%s)", row['codigoEspecificoProducto'], row['nombreProducto'])
codigoEspecificoProducto = row['codigoEspecificoProducto']
nombreProducto = row['nombreProducto']
values = (codigoEspecificoProducto,nombreProducto)
cursor.execute(query,values)
connStr.commit()
cursor.close()
connStr.close()

Transfering csv files into hdfs, with converting them to avro, using flume

I am new to Big Data and I have task to transfer csv files to HDFS using Flume, but it also should convert those csv to avro. I tried to do that using following flume configuration:
a1.channels = dataChannel
a1.sources = dataSource
a1.sinks = dataSink
a1.channels.dataChannel.type = memory
a1.channels.dataChannel.capacity = 1000000
a1.channels.dataChannel.transactionCapacity = 10000
a1.sources.dataSource.type = spooldir
a1.sources.dataSource.spoolDir = {spool_dir}
a1.sources.dataSource.fileHeader = true
a1.sources.dataSource.fileHeaderKey = file
a1.sources.dataSource.basenameHeader = true
a1.sources.dataSource.basenameHeaderKey = basename
a1.sources.dataSource.interceptors.attach-schema.type = static
a1.sources.dataSource.interceptors.attach-schema.key = flume.avro.schema.url
a1.sources.dataSource.interceptors.attach-schema.value = {path_to_schema_in_hdfs}
a1.sinks.dataSink.type = hdfs
a1.sinks.dataSink.hdfs.path = {sink_path}
a1.sinks.dataSink.hdfs.format = text
a1.sinks.dataSink.hdfs.inUsePrefix = .
a1.sinks.dataSink.hdfs.filePrefix = drone
a1.sinks.dataSink.hdfs.fileSuffix = .avro
a1.sinks.dataSink.hdfs.rollSize = 180000000
a1.sinks.dataSink.hdfs.rollCount = 100000
a1.sinks.dataSink.hdfs.rollInterval = 120
a1.sinks.dataSink.hdfs.idleTimeout = 3600
a1.sinks.dataSink.hdfs.fileType = DataStream
a1.sinks.dataSink.serializer = avro_event
The output where avro file with flume's default schema.I also tried to use AvroEventSerializer, but I just got a lot of different error, I solved all of them, except this one:
ERROR hdfs.HDFSEventSink: process failed
java.lang.ExceptionInInitializerError
at org.apache.hadoop.hdfs.DFSOutputStream.computePacketChunkSize(DFSOutputStream.java:1305)
at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1243)
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1266)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1101)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1059)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:232)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:75)
Thank you for any help.
Sory for mistakes in the config. I fixed them and found the way to convert css to avro. I a little bit modified AvroEventSerializer this way:
public void write(Event event) throws IOException {
if (dataFileWriter == null) {
initialize(event);
}
String[] items = new String(event.getBody()).split(",");
city.put("deviceID", Long.parseLong(items[0]));
city.put("groupID", Long.parseLong(items[1]));
city.put("timeCounter", Long.parseLong(items[2]));
city.put("cityCityName", items[3]);
city.put("cityStateCode", items[4]);
city.put("sessionCount", Long.parseLong(items[5]));
city.put("errorCount", Long.parseLong(items[6]));
dataFileWriter.append(citi);
}
and here is city definition:
private GenericRecord city = null;
Please reply, if you know a better way to do that

Load 3D model in Unity using Resource folder and Mysql

I want to load 3D model using Resource folder. I created an sql database to store the address. In this case I stored the file "deer-3ds" in folder "Models" and also save these information in a table named "modeladdress" in sql.
So please help me to correct my code. I know that it's 100% wrong but I dont know how to fix it. Thank you.
using UnityEngine;
using System.Collections;
using System;
using System.Data;
using Mono.Data.Sqlite;
public class addobject : MonoBehaviour {
// Use this for initialization
void Start () {
//GameObject deer=Instantiate(Resources.Load("deer-3d.bak",typeof(GameObject)))as GameObject;
// GameObject instance = Instantiate(Resources.Load("Models/deer-3ds", typeof(GameObject))) as GameObject;
string conn = "URI=file:" + Application.dataPath + "/modeladdress.s3db"; //Path to database.
IDbConnection dbconn;
dbconn = (IDbConnection) new SqliteConnection(conn);
dbconn.Open(); //Open connection to the database.
IDbCommand dbcmd = dbconn.CreateCommand();
string sqlQuery = "SELECT ordinary,foldername, filename " + "FROM modeladdress";
dbcmd.CommandText = sqlQuery;
IDataReader reader = dbcmd.ExecuteReader();
while (reader.Read ()) {
int ordinary = reader.GetInt32 (0);
string foldername = reader.GetString (1);
string filename = reader.GetString (2);
string path = foldername + "/" + filename;
//Debug.Log( "value= "+value+" name ="+name+" random ="+ rand);
GameObject instance = Instantiate(Resources.Load(path, typeof(GameObject))) as GameObject;
instance.SetActive (true);
}
reader.Close();
reader = null;
dbcmd.Dispose();
dbcmd = null;
dbconn.Close();
dbconn = null;
}
// Update is called once per frame
void Update () {
// GameObject instance = Instantiate(Resources.Load("Models/deer-3ds", typeof(GameObject))) as GameObject;
// instance.SetActive (true);
}
}
First of all, you are using SQLite at your database management system, not MySQL. Second, the way you have written your query,
string sqlQuery = "SELECT ordinary,foldername, filename " + "FROM modeladdress";
Will return the ordinary, foldername, and filename for every model. You need to use a WHERE clause to specify precisely which model you want to use. Thus, you need some way to know which model you want to query from the database before you actually execute the query, and in that case, why even query a database? You're going to have to store some unique identifier anyway so a database solves nothing.
Now concerning the actual code you have written, it appears to be correct (i.e. it should be returning what you want). The problem must be that either your table is empty, your values that are returned are incorrect, or that the object is being instantiated in an incorrect location and thus you are thinking it's not working. If you want a more concrete answer you'll have to comment on this answer with the specific problem you are facing (i.e. what specifically is "wrong"?).

System.Data.OleDb.OleDbException: Invalid path or file name

i have the following code which has been getting me data from flat files. but now all of a sudden i am getting this error
System.Data.OleDb.OleDbException: Invalid path or file name
but the code hasnt changed it worked for months,im not sure what went wrong.
System.Web.Script.Serialization.JavaScriptSerializer json = new System.Web.Script.Serialization.JavaScriptSerializer();
string jsonText;
System.Collections.Generic.List<object> objList = new List<object>();
string strConn = #"Provider=vfpoledb;Data Source=\\10.0.0.0\wwwroot\apps\assembly\FlatDatabaseDbfs\vt_Flat.dbf;Collating Sequence=machine;";
using (System.Data.OleDb.OleDbConnection conn = new System.Data.OleDb.OleDbConnection(strConn))
{
System.Data.OleDb.OleDbCommand cmddbf = new System.Data.OleDb.OleDbCommand();
cmddbf.Connection = conn;
conn.Open();
cmddbf.CommandText = "select * from vt_Flat";
var dr = cmddbf.ExecuteReader();
while (dr.Read())
{
objList.Add(new
{
Code = (dr["dp_code"].ToString().Trim()),
});
};
}
var filteredList = objList.Where(obj => ((dynamic)obj).Status == (Request.QueryString["Status"] ?? "") && ((dynamic)obj).DepCode == (Request.QueryString["Code"] ?? ""));
jsonText = json.Serialize(filteredList);
Response.Write(jsonText);
}
is there something wrong with iis permissions?
Aside from the connection having to point to the PATH as already noted by Oleg, in the C# instances of OleDbConnection I have done in the past, the connection string uses
Provider=VFPOLEDB.1
Don't know if it is case/sensitive issue and the ".1" which is also part of the provider string.
Once you have a valid connection to the PATH, then your query can query from any table within the path location. So if you had 2+ files, and needed to join them, you would do so with a standard query / join. In your case, your command text is only "select *" since you changed your original connection that included the table. Change the command text to
"select * from vt_Flat"
OTHER CONSIDERATIONS
Is this being run from some web service project? If so, THAT could be the basis. You as a developer testing are running with your permissions / access. If running as a web server, the WEB-based user account may not have permissions to the folder to process / work with the data.
Check the folder of your production data to ALLOW the web user if so running. If that doesn't work, set permissions on the folder to EVERYBODY (only for testing/confirmation purposes only). See if that is the problem.
Also, from the Provider connection, did you try with it as all upper case VFPOLEDB.1?
Use path instead of file name, e.g.:
Data Source=\\10.0.0.0\wwwroot\apps\assembly\FlatDatabaseDbfs\;

Error of update(conn,tablename,colnames,data,whereClause) by using matlab connect ODBC and mySQL server 5.6

Isn't my coding typing wrong way? I need create an update button so user can edit the information by using Matlab. After update, the button need connect to mySQL server 5.6 and ODBC connector.
This is my code:
% --- Executes on button press in update.
function update_Callback(hObject, eventdata, handles)
% hObject handle to update (see GCBO)
% eventdata reserved - to be defined in a future version of MATLAB
% handles structure with handles and user data (see GUIDATA)
%Display dialog box to confirm save
choice = questdlg('Confirm update to database?', ...
'', ...
'Yes','No','Yes');
% Handle dialog box response
switch choice
case 'Yes'
%Set preferences with setdbprefs.
setdbprefs('DataReturnFormat', 'cellarray');
%Make connection to database.
conn = database('animal_cbir', '', '');
%Test if database connection is valid
testConnection = isconnection(conn);
disp(testConnection);
fileID = getappdata(0,'namevalue');
imageID = fileID;
name = get(handles.edit11,'String');
commonName = get(handles.edit1,'String');
scientificName = get(handles.edit2,'String');
class = get(handles.edit3,'String');
diet = get(handles.edit4,'String');
habitat = get(handles.edit5,'String');
lifeSpan = get(handles.edit6,'String');
size = get(handles.edit7,'String');
weight = get(handles.edit8,'String');
characteristic = get(handles.edit10,'String');
tablename = 'animal';
colnames ={'imageID','name','commonName','scientificName','class','diet','habitat','lifeSpan','size','weight','characteristic'};
data = {imageID,name,commonName,scientificName,class,diet,habitat,lifeSpan,size,weight,characteristic};
disp (data);
whereClause = sprintf(['where imageID = "%s"'],fileID);
update(conn,tablename,colnames,data,whereClause);
updateSuccess = helpdlg('Existing animal species successfully updated in database.');
commit(conn);
case 'No'
end
Error I am getting:
No method 'setInt' with matching signature found for class 'sun.jdbc.odbc.JdbcOdbcPreparedStatement'.
Hope that anyone can help me solve it.