I'm trying to grab data from a MySQL database.
Approach 2 - apply/map style
I'm using the MySQL ADO Reference to try to build this system. In particular, the example found at 21.2.3.1.7.
(using a pseudo code)
let table = build_sequence(query.read)
Where query.read returns a row in the table(Or rather, a list of elements that happen to be a row in the table). And the table variable is a list of lists that will represent a table returned from the query.
I've stared at the code given below, and it's syntax is over my head, I'm afraid.
Approach 1 - looping.
Problem 1: It's inelegant, requiring a mutable.
Problem 2: This just feels wrong, based on my prior experience with Prolog & Lisp. There's gotta be a more...functional way to do this.
I'm not sure where to begin though. Comments & thoughts?
let reader : MySql.Data.MySqlClient.MySqlDataReader = command.ExecuteReader()
let arr = []
let mutable rowIter = 0
let readingLoop() =
while(reader.Read()) do
rowIter = rowIter + 1
for i = 0 to reader.FieldCount do
//set arr[someiterator, i] = reader.GetValue[i].ToString())
The Seq type has a neat function for handling database cursors called generate_using (see F# Manual and the Data Access chapter in Foundations of F#). This is a higher order function that takes one function to open the cursor and another (called repeatedly) to process records from the cursor. Here is some code that uses generate_using to execute a sql query:
let openConnection (connectionName : string) =
let connectionSetting = ConfigurationManager.ConnectionStrings.Item(connectionName)
let connectionString = connectionSetting.ConnectionString
let connection = new OracleConnection(connectionString)
connection.Open()
connection
let generator<'a> (reader : IDataReader) =
if reader.Read() then
let t = typeof<'a>
let props = t.GetProperties()
let types = props
|> Seq.map (fun x -> x.PropertyType)
|> Seq.to_array
let cstr = t.GetConstructor(types)
let values = Array.create reader.FieldCount (new obj())
reader.GetValues(values) |> ignore
let values = values
|> Array.map (fun x -> match x with | :? DBNull -> null | _ -> x)
Some (cstr.Invoke(values) :?> 'a)
else
None
let executeSqlReader<'a> (connectionName : string) (sql : string) : 'a list =
let connection = openConnection connectionName
let opener() =
let command = connection.CreateCommand(CommandText = sql, CommandType = CommandType.Text)
command.ExecuteReader()
let result = Seq.to_list(Seq.generate_using opener generator)
connection.Close()
connection.Dispose()
result
For example to list all the tables in an Oracle database we need to define a column definition type and invoke executeSqlReader as follows:
type ColumnDefinition = {
TableName : string;
ColumnName : string;
DataType : string;
DataLength : decimal;
}
let tableList = executeSqlReader<ColumnDefinition>
"MyDatabase"
"SELECT t.table_name, column_name, data_type, data_length FROM USER_TABLES t, USER_TAB_COLUMNS c where t.TABLE_NAME = c.table_name order by t.table_name, c.COLUMN_NAME"
It can be hard to work with imperative APIs in a non-imperative way. I don't have MySql handy, but I made an approxmiation, hopefully this will provide inspiration. Seq.unfold is a function people find pretty awesome once they grok it. List.init (or Array.init) are also handy for initializing known-size data structures without using mutables.
#light
type ThingLikeSqlReader() =
let mutable rowNum = 0
member this.Read() =
if rowNum > 3 then
false
else
rowNum <- rowNum + 1
true
member this.FieldCount = 5
member this.GetValue(i) = i + 1
let reader = new ThingLikeSqlReader()
let data = reader |> Seq.unfold (fun (reader : ThingLikeSqlReader) ->
if reader.Read() then
Some (List.init reader.FieldCount (fun i -> reader.GetValue(i)), reader)
else
None) |> Seq.to_list
printfn "%A" data
Related
I'm trying to learn F# and I feel like I can write / rewrite this block of code to be more "idiomatic" F# but I just can't figure out how I can accomplish it.
My simple program will be loading values from 2 csv files: A list of Skyrim potion effects, and a list of Skyrim Ingredients. An ingredient has 4 Effects. Once I have the Ingredients, I can write something to process them - right now, I just want to write the CSV load in a way that makes sense.
Code
Here are my types:
type Effect(name:string, id, description, base_cost, base_mag, base_dur, gold_value) =
member this.Name = name
member this.Id = id
member this.Description = description
member this.Base_Cost = base_cost
member this.Base_Mag = base_mag
member this.Base_Dur = base_dur
member this.GoldValue = gold_value
type Ingredient(name:string, id, primary, secondary, tertiary, quaternary, weight, value) =
member this.Name = name
member this.Id = id
member this.Primary = primary
member this.Secondary = secondary
member this.Tertiary = tertiary
member this.Quaternary = quaternary
member this.Weight = weight
member this.Value = value
Here is where I parse an individual comma-separated string, per type:
let convertEffectDataRow (csvLine:string) =
let cells = List.ofSeq(csvLine.Split(','))
match cells with
| name::id::effect::cost::mag::dur::value::_ ->
let effect = new Effect(name, id, effect, Decimal.Parse(cost), Int32.Parse(mag), Int32.Parse(dur), Int32.Parse(value))
Success effect
| _ -> Failure "Incorrect data format!"
let convertIngredientDataRow (csvLine:string) =
let cells = List.ofSeq(csvLine.Split(','))
match cells with
| name::id::primary::secondary::tertiary::quaternary::weight::value::_ ->
Success (new Ingredient(name, id, primary, secondary, tertiary, quaternary, Decimal.Parse(weight), Int32.Parse(value)))
| _ -> Failure "Incorrect data format!"
So I feel like I should be able to build a function that accepts one of these functions or chains them or something, so that I can recursively go through the lines in the CSV file and pass those lines to the correct function above. Here is what I've tried so far:
type csvTypeEnum = effect=1 | ingredient=2
let rec ProcessStuff lines (csvType:csvTypeEnum) =
match csvType, lines with
| csvTypeEnum.effect, [] -> []
| csvTypeEnum.effect, currentLine::remaining ->
let parsedLine = convertEffectDataRow2 currentLine
let parsedRest = ProcessStuff remaining csvType
parsedLine :: parsedRest
| csvTypeEnum.ingredient, [] -> []
| csvTypeEnum.ingredient, currentLine::remaining ->
let parsedLine = convertIngredientDataRow2 currentLine
let parsedRest = ProcessStuff remaining csvType
parsedLine :: parsedRest
| _, _ -> Failure "Error in pattern matching"
But this (predictably) has a compile error on second instance of recursion and the last pattern. Specifically, the second time parsedLine :: parsedRest shows up does not compile. This is because the function is attempting to both return an Effect and an Ingredient, which obviously won't do.
Now, I could just write 2 entirely different functions to handle the different CSVs, but that feels like extra duplication. This might be a harder problem than I'm giving it credit for, but it feels like this should be rather straightforward.
Sources
The CSV parsing code I took from chapter 4 of this book: https://www.manning.com/books/real-world-functional-programming
Since the line types aren't interleaved into the same file and they refer to different csv file formats, I would probably not go for a Discriminated Union and instead pass the processing function to the function that processes the file line by line.
In terms of doing things idiomatically, I would use a Record rather than a standard .NET class for this kind of simple data container. Records provide automatic equality and comparison implementations which are useful in F#.
You can define them like this:
type Effect = {
Name : string; Id: string; Description : string; BaseCost : decimal;
BaseMag : int; BaseDuration : int; GoldValue : int
}
type Ingredient= {
Name : string; Id: string; Primary: string; Secondary : string; Tertiary : string;
Quaternary : string; Weight : decimal; GoldValue : int
}
That requires a change to the conversion function, e.g.
let convertEffectDataRow (csvLine:string) =
let cells = List.ofSeq(csvLine.Split(','))
match cells with
| name::id::effect::cost::mag::dur::value::_ ->
Success {Name = name; Id = id; Description = effect; BaseCost = Decimal.Parse(cost);
BaseMag = Int32.Parse(mag); BaseDuration = Int32.Parse(dur); GoldValue = Int32.Parse(value)}
| _ -> Failure "Incorrect data format!"
Hopefully it's obvious how to do the other one.
Finally, cast aside the enum and simply replace it with the appropriate line function (I've also swapped the order of the arguments).
let rec processStuff f lines =
match lines with
|[] -> []
|current::remaining -> f current :: processStuff f remaining
The argument f is just a function that is applied to each string line. Suitable f values are the functions we created above, e.g.convertEffectDataRow. So you can simply call processStuff convertEffectDataRow to process an effect file and processStuff convertIngredientDataRow to process and ingredients file.
However, now we've simplified the processStuff function, we can see it has type: f:('a -> 'b) -> lines:'a list -> 'b list. This is the same as the built-in List.map function so we can actually remove this custom function entirely and just use List.map.
let processEffectLines lines = List.map convertEffectDataRow lines
let processIngredientLines lines = List.map convertIngredientDataRow lines
(optional) Convert Effect and Ingredient to records, as s952163 suggested.
Think carefully about the return types of your functions. ProcessStuff returns a list from one case, but a single item (Failure) from the other case. Thus compilation error.
You haven't shown what Success and Failure definitions are. Instead of generic success, you could define the result as
type Result =
| Effect of Effect
| Ingredient of Ingredient
| Failure of string
And then the following code compiles correctly:
let convertEffectDataRow (csvLine:string) =
let cells = List.ofSeq(csvLine.Split(','))
match cells with
| name::id::effect::cost::mag::dur::value::_ ->
let effect = new Effect(name, id, effect, Decimal.Parse(cost), Int32.Parse(mag), Int32.Parse(dur), Int32.Parse(value))
Effect effect
| _ -> Failure "Incorrect data format!"
let convertIngredientDataRow (csvLine:string) =
let cells = List.ofSeq(csvLine.Split(','))
match cells with
| name::id::primary::secondary::tertiary::quaternary::weight::value::_ ->
Ingredient (new Ingredient(name, id, primary, secondary, tertiary, quaternary, Decimal.Parse(weight), Int32.Parse(value)))
| _ -> Failure "Incorrect data format!"
type csvTypeEnum = effect=1 | ingredient=2
let rec ProcessStuff lines (csvType:csvTypeEnum) =
match csvType, lines with
| csvTypeEnum.effect, [] -> []
| csvTypeEnum.effect, currentLine::remaining ->
let parsedLine = convertEffectDataRow currentLine
let parsedRest = ProcessStuff remaining csvType
parsedLine :: parsedRest
| csvTypeEnum.ingredient, [] -> []
| csvTypeEnum.ingredient, currentLine::remaining ->
let parsedLine = convertIngredientDataRow currentLine
let parsedRest = ProcessStuff remaining csvType
parsedLine :: parsedRest
| _, _ -> [Failure "Error in pattern matching"]
csvTypeEnum type looks fishy, but I'm not sure what you were trying to achieve, so just fixed the compilation errors.
Now you can refactor your code to reduce duplication by passing functions as parameters when needed. But always start with types!
You can certainly pass a function to another function and use a DU as a return type, for example:
type CsvWrapper =
| CsvA of string
| CsvB of int
let csvAfunc x =
CsvA x
let csvBfunc x =
CsvB x
let csvTopFun x =
x
csvTopFun csvBfunc 5
csvTopFun csvAfunc "x"
As for the type definitions, you can just use records, will save you some typing:
type Effect = {
name:string
id: int
description: string
}
let eff = {name="X";id=9;description="blah"}
I want to use a generic query function like:
let filter predicateFn =
predicateFn |>
fun f s -> query { for x in s do
where (f(x))
select x }
Where f should be a predicate function. Obviously, this cannot be translated to a sql query. But is there a solution for this problem. I have seen sample solutions for C#.
Approach taken from: Web, Cloud & Mobile Solutions with F# by Daniel Mohl.
Edit: example to clarify:
let userSorter = fun (u:users) -> u.Login
let sorted =
query { for user in dbConn.GetDataContext().Users do
sortBy ((fun (u:users) -> u.Login)(user))
select user.Login }
Basically, I want to replace the lambda by the userSorter. As it is the above code runs, but this will fail:
let userSorter = fun (u:users) -> u.Login
let sorted =
query { for user in dbConn.GetDataContext().Users do
sortBy (userSorter(user))
select user.Login }
Finally found the solution, already on Stackoverflow:
open System.Linq
open Microsoft.FSharp.Quotations
open Microsoft.FSharp.Quotations.Patterns
let runQuery (q:Expr<IQueryable<'T>>) =
match q with
| Application(Lambda(builder, Call(Some builder2, miRun, [Quote body])), queryObj) ->
query.Run(Expr.Cast<Microsoft.FSharp.Linq.QuerySource<'T, IQueryable>>(body))
| _ -> failwith "Wrong argument"
let test () =
<# query { for p in db.Products do
where ((%predicate) p)
select p.ProductName } #>
|> runQuery
|> Seq.iter (printfn "%s")
Credits should go to Tomas Petricek, you can have my bounty!
Edit: To match against any query result of type 'T see my other Q&A
I have a NameValueCollection which I need to convert to a Map and I just can't work it out. I tried:
let headerMap (m : MailMessage) = m.Headers |> Map.map (fun k v -> v.[k])
Do I need to use Seq.map instead?
Basically the point of this is that I want to serialize the headers in a System.Net.MailMessage to JSON.
Daniel's answer will work just fine, but I thought I'd offer some additional alternatives:
Array.fold -- This should be faster than Daniel's version since it avoids the overhead of the iterators.
let mapOfNameValueCollection (collection : NameValueCollection) =
(Map.empty, collection.AllKeys)
||> Array.fold (fun map key ->
let value = collection.[key]
Map.add key value map)
Array.fold with sets of values -- Similar to the code above, but returns the value as a Set<string> which may be useful if you want to determine if some value is in the returned set of values.
let mapOfNameValueCollection (collection : NameValueCollection) =
(Map.empty, collection.AllKeys)
||> Array.fold (fun map key ->
let valueSet =
match collection.[key] with
| null ->
Set.empty
| values ->
Set.ofArray <| values.Split [| ',' |]
Map.add key valueSet map)
Recursive loop -- Creates the map item-by-item with a recursive loop. I wouldn't use this in practice because the Array.fold version would be easier and faster. However, this approach could be faster if the specific collection class you're using (derived from NameValueCollection) overrides the AllKeys property and has some weird internal behavior which takes a long time to return the property value.
let mapOfNameValueCollection (collection : NameValueCollection) =
let rec createMap map idx =
if idx < 0 then map
else
let itemName = collection.GetKey idx
let itemValue = collection.[itemName]
let map = Map.add itemName itemValue map
createMap map (idx - 1)
createMap Map.empty (collection.Count - 1)
Imperative loop -- Creates the map item-by-item with an imperative loop. As with the recursive loop, I'd prefer to use Array.fold in practice unless there was some special reason not to.
let mapOfNameValueCollection (collection : NameValueCollection) =
let mutable map = Map.empty
let maxIndex = collection.Count - 1
for i = 0 to maxIndex do
let itemName = collection.GetKey i
let itemValue = collection.[itemName]
map <- Map.add itemName itemValue map
map
nvc.AllKeys
|> Seq.map (fun key -> key, nvc.[key])
|> Map.ofSeq
here's the problem. I need to time a function in f# using another function. I have this piece of code
let time f a =
let start = System.DateTime.Now in
let res = (fun f a -> f(a)) in
let finish = System.DateTime.Now in
(res, finish - start)
which I'm trying to call saying
time ackermann (2,9);;
I have a function ackermann that takes a tuple (s,n) as argument
Probably something fundamentally wrong with this but I don't think I'm far away from a solution that could and looks somewhat like this.
Any suggestions?
Oh btw. the error message I'm getting is saying :
stdin(19,1): error FS0030: Value restriction. The value 'it' has been inferred to have generic type
val it : (('_a -> '_b) -> '_a -> '_b) * System.TimeSpan
Either define 'it' as a simple data term, make it a function with explicit arguments or, if you do not intend for it to be generic, add a type annotation.
You have at least two issues:
Try let res = f a. You already have values f and a in scope, but you're currently defining res as a function which takes a new f and applies it to a new a.
Don't use DateTimes (which are appropriate for representing dates and times, but not short durations). Instead, you should be using a System.Diagnostics.Stopwatch.
You can do something like this:
let time f =
let sw = System.Diagnostics.Stopwatch.StartNew()
let r = f()
sw.Stop()
printfn "%O" sw.Elapsed
r
Usage
time (fun () -> System.Threading.Thread.Sleep(100))
I usually keep the following in my code files when sending a bunch of stuff to fsi.
#if INTERACTIVE
#time "on"
#endif
That turns on fsi's built-in timing, which provides more than just execution time:
Real: 00:00:00.099, CPU: 00:00:00.000, GC gen0: 0, gen1: 0, gen2: 0
I would do it like this:
let time f = fun a ->
let start = System.DateTime.Now
let res = f a
(res, System.DateTime.Now - start)
You can then use it to create timed functions e.g.
let timedAckermann = time ackermann
let (res, period) = timedAckermann (2,9)
You should also consider using System.Diagnostics.Stopwatch for timing instead of DateTimes.
As was already suggested, you should use Stopwatch instead of DateTime for this kind of timing analyses.
What wasn't mentioned yet is that if you for some reason need to use DateTime, then always consider using DateTime.UtcNow rather than DateTime.Now. The implementation of DateTime.Now can be paraphrased as "DateTime.UtcNow.ToLocalTime()", and that "ToLocalTime()" part is doing more than you might think it would do. In addition to having less overhead, DateTime.UtcNow also avoids headaches related to daylight savings time. You can find several articles and blog posts on the web on the differences between DateTime.Now and DateTime.UtcNow
Inspired by how FSharp Interactive does it (see https://github.com/Microsoft/visualfsharp/blob/master/src/fsharp/fsi/fsi.fs#L175), this will time the function plus report how much CPU, allocation, etc.
Example output: Real: 00:00:00.2592820, CPU: 00:00:26.1814902, GC gen0: 30, gen1: 1, gen2: 0
let time f =
let ptime = System.Diagnostics.Process.GetCurrentProcess()
let numGC = System.GC.MaxGeneration
let startTotal = ptime.TotalProcessorTime
let startGC = [| for i in 0 .. numGC -> System.GC.CollectionCount(i) |]
let stopwatch = System.Diagnostics.Stopwatch.StartNew()
let res = f ()
stopwatch.Stop()
let total = ptime.TotalProcessorTime - startTotal
let spanGC = [ for i in 0 .. numGC-> System.GC.CollectionCount(i) - startGC.[i] ]
let elapsed = stopwatch.Elapsed
printfn "Real: %A, CPU: %A, GC %s" elapsed total ( spanGC |> List.mapi (sprintf "gen%i: %i") |> String.concat ", ")
res
I would like to transfer a SQL table (let say i. 2 columns : one containing users ID and one containing users age and ii. n rows) containing only integers into a F# matrix (same dimensions).
I manage to do so with the following F# code, but I am convinced it is not the most efficient way to do so.
Indeed, the only way I found to dimensionate the F# matrix was to create 2 tables with a single value (number of rows and number of columns respectively) using MySQL and transfer these value into F#.
Is it possible to import a mySQL table into a F# matrix with a F# code which "recognize" the dimension of the matrix. Basically I would like a function which take a table address as an argument and return a matrix.
Here is my code :
#r "FSharp.PowerPack.dll"
#r "Microsoft.Office.Interop.Excel"
open System
open System.Data
open System.Data.SqlClient
open Microsoft.Office.Interop
open Microsoft.FSharp.Math
open System.Collections.Generic
//Need of three types : User, number of rows and number of columns
type user = {
ID : int;
Age : int;}
type nbrRows = {NbreL : int ;}
type nbrCol = {NbreC : int ;}
// I. Import the SQL data into F#
// I.1. Import the number of rows of the table into F#
let NbrRows = seq {
use cnn = new SqlConnection(#"myconnection; database=MyDataBase; integrated security=true")
use cmd1 = new SqlCommand("Select * from theTablesWhichContainsTheNumberOfRows", cnn)
cnn.Open()
use reader = cmd1.ExecuteReader()
while reader.Read() do
yield {
NbreL = unbox(reader.["Expr1"])
}
}
let NbrRowsList = NbrRows |> Seq.toList // convert the sequence into a List
// I.2. Same code to import the number of columns of the table
let NbrCol = seq {
use cnn = new SqlConnection(#"MyConnection; database=myDatabase; integrated security=true")
use cmd1 = new SqlCommand("Select * from theTablesWhichContainsTheNumberOfColumns", cnn)
cnn.Open()
use reader = cmd1.ExecuteReader()
while reader.Read() do
yield {
NbreC = unbox(reader.["Expr1"])
}
}
let NbrColsList = NbrCol |> Seq.toList
// Initialisation of the Matrix
let matrixF = Matrix.create NbrRowsList.[0].NbreL NbrColsList.[0].NbreC 0.
//Transfer of the mySQL User table into F# through a sequence as previously
let GetUsers = seq {
use cnn = new SqlConnection(#"myConnection, database=myDatabase; integrated security=true")
use cmd = new SqlCommand("Select * from tableUser ORDER BY ID", cnn)
cnn.Open()
use reader = cmd.ExecuteReader()
while reader.Read() do
yield {
ID = unbox(reader.["ID"])
Age = unbox(reader.["Age"])
}
}
// Sequence to list
let UserDatabaseList = GetUsers |> Seq.toList
// Fill of the user matrix
for i in 0 .. (NbrRowList.[0].NbreL - 1) do
matrixF.[0,i] <- UserDatabaseList.[i].ID |> float
matrixF.[1,i] <- UserDatabaseList.[i].Age|> float
matrixUsers
There are various ways to initialize matrix if you don't know its size in advance. For example Matrix.ofList takes a list of lists and it calculates the size automatically.
If you have just UserDatabaseList (which you can create withtout knowing the number of rows and columns), then you should be able to write:
Matrix.ofList
[ // Create list containing rows from the database
for row in UserDatabaseList do
// For each row, return list of columns (float values)
yield [ float row.ID; float row.Age ] ]
Aside - F# matrix is really useful mainly if you're going to do some matrix operations (and even then, it is not the most efficient option). If you're doing some data processing, then it may be easier to keep the data in an ordinary list. If you're doing some serious math, then you may want to check how to use Math.NET library from F#, which has more efficient matrix type.