Transfer a MySQL table into a F# matrix - mysql

I would like to transfer a SQL table (let say i. 2 columns : one containing users ID and one containing users age and ii. n rows) containing only integers into a F# matrix (same dimensions).
I manage to do so with the following F# code, but I am convinced it is not the most efficient way to do so.
Indeed, the only way I found to dimensionate the F# matrix was to create 2 tables with a single value (number of rows and number of columns respectively) using MySQL and transfer these value into F#.
Is it possible to import a mySQL table into a F# matrix with a F# code which "recognize" the dimension of the matrix. Basically I would like a function which take a table address as an argument and return a matrix.
Here is my code :
#r "FSharp.PowerPack.dll"
#r "Microsoft.Office.Interop.Excel"
open System
open System.Data
open System.Data.SqlClient
open Microsoft.Office.Interop
open Microsoft.FSharp.Math
open System.Collections.Generic
//Need of three types : User, number of rows and number of columns
type user = {
ID : int;
Age : int;}
type nbrRows = {NbreL : int ;}
type nbrCol = {NbreC : int ;}
// I. Import the SQL data into F#
// I.1. Import the number of rows of the table into F#
let NbrRows = seq {
use cnn = new SqlConnection(#"myconnection; database=MyDataBase; integrated security=true")
use cmd1 = new SqlCommand("Select * from theTablesWhichContainsTheNumberOfRows", cnn)
cnn.Open()
use reader = cmd1.ExecuteReader()
while reader.Read() do
yield {
NbreL = unbox(reader.["Expr1"])
}
}
let NbrRowsList = NbrRows |> Seq.toList // convert the sequence into a List
// I.2. Same code to import the number of columns of the table
let NbrCol = seq {
use cnn = new SqlConnection(#"MyConnection; database=myDatabase; integrated security=true")
use cmd1 = new SqlCommand("Select * from theTablesWhichContainsTheNumberOfColumns", cnn)
cnn.Open()
use reader = cmd1.ExecuteReader()
while reader.Read() do
yield {
NbreC = unbox(reader.["Expr1"])
}
}
let NbrColsList = NbrCol |> Seq.toList
// Initialisation of the Matrix
let matrixF = Matrix.create NbrRowsList.[0].NbreL NbrColsList.[0].NbreC 0.
//Transfer of the mySQL User table into F# through a sequence as previously
let GetUsers = seq {
use cnn = new SqlConnection(#"myConnection, database=myDatabase; integrated security=true")
use cmd = new SqlCommand("Select * from tableUser ORDER BY ID", cnn)
cnn.Open()
use reader = cmd.ExecuteReader()
while reader.Read() do
yield {
ID = unbox(reader.["ID"])
Age = unbox(reader.["Age"])
}
}
// Sequence to list
let UserDatabaseList = GetUsers |> Seq.toList
// Fill of the user matrix
for i in 0 .. (NbrRowList.[0].NbreL - 1) do
matrixF.[0,i] <- UserDatabaseList.[i].ID |> float
matrixF.[1,i] <- UserDatabaseList.[i].Age|> float
matrixUsers

There are various ways to initialize matrix if you don't know its size in advance. For example Matrix.ofList takes a list of lists and it calculates the size automatically.
If you have just UserDatabaseList (which you can create withtout knowing the number of rows and columns), then you should be able to write:
Matrix.ofList
[ // Create list containing rows from the database
for row in UserDatabaseList do
// For each row, return list of columns (float values)
yield [ float row.ID; float row.Age ] ]
Aside - F# matrix is really useful mainly if you're going to do some matrix operations (and even then, it is not the most efficient option). If you're doing some data processing, then it may be easier to keep the data in an ordinary list. If you're doing some serious math, then you may want to check how to use Math.NET library from F#, which has more efficient matrix type.

Related

How to read from MySQL table Polygon data

I am developing an application that I need location data to be stored on MySQL table. In addition to point locations, I need regions (polygon) as well.
I am currently writing the polygon coordinates as follow :
oMySQLConnecion = new MySqlConnection(DatabaseConnectionString);
if (oMySQLConnecion.State == System.Data.ConnectionState.Closed || oMySQLConnecion.State == System.Data.ConnectionState.Broken)
{
oMySQLConnecion.Open();
}
if (oMySQLConnecion.State == System.Data.ConnectionState.Open)
{
string Query = #"INSERT INTO region (REGION_POLYGON) VALUES (PolygonFromText(#Parameter1))";
MySqlCommand oCommand = new MySqlCommand(Query, oMySQLConnecion);
oCommand.Parameters.AddWithValue("#Parameter1", PolygonString);
int sqlSuccess = oCommand.ExecuteNonQuery();
oMySQLConnecion.Close();
oDBStatus.Type = DBDataStatusType.SUCCESS;
oDBStatus.Message = DBMessageType.SUCCESSFULLY_DATA_UPDATED;
return oDBStatus;
}
After the execution, I see the Blob in MySQL table.
Now I want to read the data back for my testing and it does not work the way I tried below :
if (oMySQLConnecion.State == System.Data.ConnectionState.Open)
{
string Query = #"SELECT REGION_ID,REGION_NICK_NAME,GeomFromText(REGION_POLYGON) AS POLYGON FROM region WHERE REGION_USER_ID = #Parameter1";
MySqlCommand oCommand = new MySqlCommand(Query, oMySQLConnecion);
oCommand.Parameters.AddWithValue("#Parameter1", UserID);
using (var reader = oCommand.ExecuteReader())
{
while (reader.Read())
{
R_PolygonCordinates oRec = new R_PolygonCordinates();
oRec.RegionNumber = Convert.ToInt32(reader["REGION_ID"]);
oRec.RegionNickName = reader["REGION_NICK_NAME"].ToString();
oRec.PolygonCodinates = reader["POLYGON"].ToString();
polygons.Add(oRec);
}
}
int sqlSuccess = oCommand.ExecuteNonQuery();
oMySQLConnecion.Close();
return polygons;
}
It returns an empty string.
I am not sure if I am really writing the data since I can not read Blob.
Is my reading syntax incorrect?
** Note:** I am using Visual Studio 2017. The MySQL latest version with Spacial classes.
Any help is highly appreciated.
Thanks
GeomFromText() takes a WKT (the standardized "well-known text" format) value as input and returns the MySQL internal geometry type as output.
This is the inverse of what you need, which is ST_AsWKT() or ST_AsText() -- take an internal-format geometry object as input and return WKT as output.
Prior to 5.6, the function is called AsWKT() or AsText(). In 5.7 these are all synonyms for exactly the same function, but the non ST_* functions are deprecated and will be removed in the future.
https://dev.mysql.com/doc/refman/5.7/en/gis-format-conversion-functions.html#function_st-astext
I don't know for certain what the ST_ prefix means, but I assume it's "spatial type." There's some discussion in WL#8055 that may be of interest.

Load 3D model in Unity using Resource folder and Mysql

I want to load 3D model using Resource folder. I created an sql database to store the address. In this case I stored the file "deer-3ds" in folder "Models" and also save these information in a table named "modeladdress" in sql.
So please help me to correct my code. I know that it's 100% wrong but I dont know how to fix it. Thank you.
using UnityEngine;
using System.Collections;
using System;
using System.Data;
using Mono.Data.Sqlite;
public class addobject : MonoBehaviour {
// Use this for initialization
void Start () {
//GameObject deer=Instantiate(Resources.Load("deer-3d.bak",typeof(GameObject)))as GameObject;
// GameObject instance = Instantiate(Resources.Load("Models/deer-3ds", typeof(GameObject))) as GameObject;
string conn = "URI=file:" + Application.dataPath + "/modeladdress.s3db"; //Path to database.
IDbConnection dbconn;
dbconn = (IDbConnection) new SqliteConnection(conn);
dbconn.Open(); //Open connection to the database.
IDbCommand dbcmd = dbconn.CreateCommand();
string sqlQuery = "SELECT ordinary,foldername, filename " + "FROM modeladdress";
dbcmd.CommandText = sqlQuery;
IDataReader reader = dbcmd.ExecuteReader();
while (reader.Read ()) {
int ordinary = reader.GetInt32 (0);
string foldername = reader.GetString (1);
string filename = reader.GetString (2);
string path = foldername + "/" + filename;
//Debug.Log( "value= "+value+" name ="+name+" random ="+ rand);
GameObject instance = Instantiate(Resources.Load(path, typeof(GameObject))) as GameObject;
instance.SetActive (true);
}
reader.Close();
reader = null;
dbcmd.Dispose();
dbcmd = null;
dbconn.Close();
dbconn = null;
}
// Update is called once per frame
void Update () {
// GameObject instance = Instantiate(Resources.Load("Models/deer-3ds", typeof(GameObject))) as GameObject;
// instance.SetActive (true);
}
}
First of all, you are using SQLite at your database management system, not MySQL. Second, the way you have written your query,
string sqlQuery = "SELECT ordinary,foldername, filename " + "FROM modeladdress";
Will return the ordinary, foldername, and filename for every model. You need to use a WHERE clause to specify precisely which model you want to use. Thus, you need some way to know which model you want to query from the database before you actually execute the query, and in that case, why even query a database? You're going to have to store some unique identifier anyway so a database solves nothing.
Now concerning the actual code you have written, it appears to be correct (i.e. it should be returning what you want). The problem must be that either your table is empty, your values that are returned are incorrect, or that the object is being instantiated in an incorrect location and thus you are thinking it's not working. If you want a more concrete answer you'll have to comment on this answer with the specific problem you are facing (i.e. what specifically is "wrong"?).

Deedle Frame From Database, What is the Schema?

I am new to Deedle, and in documentation I cant find how to solve my problem.
I bind an SQL Table to a Deedle Frame using the following code:
namespace teste
open FSharp.Data.Sql
open Deedle
open System.Linq
module DatabaseService =
[<Literal>]
let connectionString = "Data Source=*********;Initial Catalog=*******;Persist Security Info=True;User ID=sa;Password=****";
type bd = SqlDataProvider<
ConnectionString = connectionString,
DatabaseVendor = Common.DatabaseProviderTypes.MSSQLSERVER >
type Database() =
static member contextDbo() =
bd.GetDataContext().Dbo
static member acAgregations() =
Database.contextDbo().AcAgregations |> Frame.ofRecords
static member acBusyHourDefinition() =
Database.contextDbo().AcBusyHourDefinition
|> Frame.ofRecords "alternative_reference_table_scan", "formula"]
static member acBusyHourDefinitionFilterByTimeAgregationTipe(value:int) =
Database.acBusyHourDefinition()
|> Frame.getRows
These things are working properly becuse I can't understand the Data Frame Schema, for my surprise, this is not a representation of the table.
My question is:
how can I access my database elements by Rows instead of Columns (columns is the Deedle Default)? I Thied what is showed in documentation, but unfortunatelly, the columns names are not recognized, as is in the CSV example in Deedle Website.
With Frame.ofRecords you can extract the table into a dataframe and then operate on its rows or columns. In this case I have a very simple table. This is for SQL Server but I assume MySQL will work the same. If you provide more details in your question the solution can narrowed down.
This is the table, indexed by ID, which is Int64:
You can work with the rows or the columns:
#if INTERACTIVE
#load #"..\..\FSLAB\packages\FsLab\FsLab.fsx"
#r "System.Data.Linq.dll"
#r "FSharp.Data.TypeProviders.dll"
#endif
//open FSharp.Data
//open System.Data.Linq
open Microsoft.FSharp.Data.TypeProviders
open Deedle
[<Literal>]
let connectionString1 = #"Data Source=(LocalDB)\MSSQLLocalDB;AttachDbFilename=C:\Users\userName\Documents\tes.sdf.mdf"
type dbSchema = SqlDataConnection<connectionString1>
let dbx = dbSchema.GetDataContext()
let table1 = dbx.Table_1
query { for row in table1 do
select row} |> Seq.takeWhile (fun x -> x.ID < 10L) |> Seq.toList
// check if we can connect to the DB.
let df = table1 |> Frame.ofRecords // pull the table into a df
let df = df.IndexRows<System.Int64>("ID") // if you need an index
df.GetRows(2L) // Get the second row, but this can be any kind of index/key
df.["Number"].GetSlice(Some 2L, Some 5L) // get the 2nd to 5th row from the Number column
Will get you the following output:
val it : Series<System.Int64,float> =
2 -> 2
>
val it : Series<System.Int64,float> =
2 -> 2
3 -> 3
4 -> 4
5 -> 5
Depending on what you're trying to do Selecting Specific Rows in Deedle might also work.
Edit
From your comment you appear to be working with some large table. Depending on how much memory you have and how large the table you still might be able to load it. If not these are some of things you can do in increasing complexity:
Use a query { } expression like above to narrow the dataset on the database server and convert just part of the result into a dataframe. You can do quite complex transformations so you might not even need the dataframe in the end. This is basically Linq2Sql.
Use lazy loading in Deedle. This works with series so you can get a few series and reassemble a dataframe.
Use Big Deedle which is designed for this sort of thing.

How to update an SQL table with F#

I would like to update a MySQL table from an F# query.
Basically I am now able to import a MySql table into a F# matrix.
I am doing some calculation on this matrix through F# and after the calculation I would like to update the primary MySQL table with the new values obtained.
For example let's take a simple matrix coming from a MySQL table :
let m = matrix [[1.;2.;3.];[1.;1.;3.];[3.;3.;3.]]
Now I would like to design a query which updates the mySQL table.
let query() =
seq { use conn = new SqlConnection(connString)
do conn.Open()
use comm = new SqlCommand("UPDATE MyTAble SET ID = "the first column of the matrix",
conn)
}
Do I need to convert the matrix into a sequence?
For this iterative update, do I need to use T-SQL (I found this way by reading some others answers?
There is no built-in functionality that would make it easier to update F# matrix in a database, so you'll have to write the update yourself. I think the best approach is to simply iterate over the matrix using for and run the update command if the value has changed. Something like:
let m1 = matrix [[1.;0.]; [0.;2.]] // The original value from DB
let m2 = m1 + m1 // New updated matrix
for i in 0 .. (fst m1.Dimensions) - 1 do
for j in 0 .. (snd m1.Dimensions) - 1 do
if m1.[i, j] <> m2.[i, j] then
// Here you need to do the actual update
printfn "SET %d, %d = %f" i j m2.[i, j]
Your function already shows how to run a single UPDATE command - just note that you don't need to wrap it in seq { .. } if you're not returning a sequence of results (here, you just want to perform some action).
If the number of changed values is big, then it may be useful to use some bulk update functionality (so that you don't run too many SQL commands which may be slow), but I'm not sure if MySQL has anything like that.

F# Beginner: retrieving an array of data from a server

I'm trying to grab data from a MySQL database.
Approach 2 - apply/map style
I'm using the MySQL ADO Reference to try to build this system. In particular, the example found at 21.2.3.1.7.
(using a pseudo code)
let table = build_sequence(query.read)
Where query.read returns a row in the table(Or rather, a list of elements that happen to be a row in the table). And the table variable is a list of lists that will represent a table returned from the query.
I've stared at the code given below, and it's syntax is over my head, I'm afraid.
Approach 1 - looping.
Problem 1: It's inelegant, requiring a mutable.
Problem 2: This just feels wrong, based on my prior experience with Prolog & Lisp. There's gotta be a more...functional way to do this.
I'm not sure where to begin though. Comments & thoughts?
let reader : MySql.Data.MySqlClient.MySqlDataReader = command.ExecuteReader()
let arr = []
let mutable rowIter = 0
let readingLoop() =
while(reader.Read()) do
rowIter = rowIter + 1
for i = 0 to reader.FieldCount do
//set arr[someiterator, i] = reader.GetValue[i].ToString())
The Seq type has a neat function for handling database cursors called generate_using (see F# Manual and the Data Access chapter in Foundations of F#). This is a higher order function that takes one function to open the cursor and another (called repeatedly) to process records from the cursor. Here is some code that uses generate_using to execute a sql query:
let openConnection (connectionName : string) =
let connectionSetting = ConfigurationManager.ConnectionStrings.Item(connectionName)
let connectionString = connectionSetting.ConnectionString
let connection = new OracleConnection(connectionString)
connection.Open()
connection
let generator<'a> (reader : IDataReader) =
if reader.Read() then
let t = typeof<'a>
let props = t.GetProperties()
let types = props
|> Seq.map (fun x -> x.PropertyType)
|> Seq.to_array
let cstr = t.GetConstructor(types)
let values = Array.create reader.FieldCount (new obj())
reader.GetValues(values) |> ignore
let values = values
|> Array.map (fun x -> match x with | :? DBNull -> null | _ -> x)
Some (cstr.Invoke(values) :?> 'a)
else
None
let executeSqlReader<'a> (connectionName : string) (sql : string) : 'a list =
let connection = openConnection connectionName
let opener() =
let command = connection.CreateCommand(CommandText = sql, CommandType = CommandType.Text)
command.ExecuteReader()
let result = Seq.to_list(Seq.generate_using opener generator)
connection.Close()
connection.Dispose()
result
For example to list all the tables in an Oracle database we need to define a column definition type and invoke executeSqlReader as follows:
type ColumnDefinition = {
TableName : string;
ColumnName : string;
DataType : string;
DataLength : decimal;
}
let tableList = executeSqlReader<ColumnDefinition>
"MyDatabase"
"SELECT t.table_name, column_name, data_type, data_length FROM USER_TABLES t, USER_TAB_COLUMNS c where t.TABLE_NAME = c.table_name order by t.table_name, c.COLUMN_NAME"
It can be hard to work with imperative APIs in a non-imperative way. I don't have MySql handy, but I made an approxmiation, hopefully this will provide inspiration. Seq.unfold is a function people find pretty awesome once they grok it. List.init (or Array.init) are also handy for initializing known-size data structures without using mutables.
#light
type ThingLikeSqlReader() =
let mutable rowNum = 0
member this.Read() =
if rowNum > 3 then
false
else
rowNum <- rowNum + 1
true
member this.FieldCount = 5
member this.GetValue(i) = i + 1
let reader = new ThingLikeSqlReader()
let data = reader |> Seq.unfold (fun (reader : ThingLikeSqlReader) ->
if reader.Read() then
Some (List.init reader.FieldCount (fun i -> reader.GetValue(i)), reader)
else
None) |> Seq.to_list
printfn "%A" data