github.com/ory/ladon is a library to manage role based access, written in golang. It contains a manager that is supposed to persist policies in the database and work with the database. The manager works fine for the in-memory case. When I use the manager to interface with sql, the required tables are not getting created.
db, err := sqlx.Open("mysql", "tx81:#tcp(127.0.0.1:3306)/policies")
......
err=db.Ping()
if err == nil {
fmt.Printf("Database is up")
}
warden := ladon.Ladon{
Manager: manager.NewSQLManager(db, nil),
}
var pol = &ladon.DefaultPolicy{
......
}
err = warden.Manager.Create(pol)
fmt.Printf("%s", err)
The error is printed as:
Table 'policies.ladon_policy' doesn't exist.
Why aren't the tables getting created?
Resources: https://github.com/ory/ladon#persistence
You need to call manager.CreateSchemas, which is never done by the manager itself.
You provide it with the schema name (postgresql only I believe) and table name to keep track of migration info.
eg:
if num, err := m.CreateSchemas("", "my_migration_table"); err != nil {
panic(err)
} else {
log.Infof("ran %d migrations", num)
}
It could probably use more documentation, you may want to file an issue with the author.
Related
After connecting to a Planetscale database using DSN with the following structure:
user:password#tcp(host)/database?tls=true
I decided to test it by running a simple query. Here's my main.go file:
package main
import (
"database/sql"
"fmt"
"os"
_ "github.com/go-sql-driver/mysql"
)
func GetDatabase() (*sql.DB, error) {
db, err := sql.Open("mysql", os.Getenv("DSN"))
return db, err
}
func main() {
db, err := GetDatabase()
if err != nil {
panic(err)
}
if err := db.Ping(); err != nil {
panic(err)
}
// ---
query, err := db.Query("SELECT name FROM status;")
if err != nil {
panic(err.Error())
}
var name string
for query.Next() {
err = query.Scan(&name)
if err != nil {
panic(err.Error())
}
fmt.Println(name)
}
// ---
fmt.Println("Successfully connected to PlanetScale!")
}
If I remove the section between the two comment lines, it will print out the 'successfully connected' message; however, if I try to run it including the query part, the outcome changes to:
panic: Error 1046: No database selected
Well, I decided to try with the query:
"USE database; SELECT name FROM status;"
However, the default driver won't take multiple statements and returns syntax error.
I know it seems like a simple question, but I already read the documentation from planetscale, from the go mysql driver, and I haven't been able to overcome this little issue. Any help will be greatly appreciated.
Looks like the problem is with the .env file. If I save the same information to a string variable within the same GetDatabase() function, and then pass it as argument instead of os.Getenv(), it connects normally. Can't tell exactly why that happens, but if someone can provide further information, I'll be thankful.
Edit:
when saving my credentials to an .env file, and using the built-in function os.Getenv("DSN"), the program will indeed connect to planetscale, but it won't select the given database in the DSN string. As mentioned in the question, selecting the database first and then running statements / queries doesn't work because of the constraints of the driver.
If I hard code the variables to a string variable, eg:
dsn := "root:mlc#tcp(127.0.0.1:3306)/app"
and then use that to open the connection, this time the program will use the desired database with any problem.
I know the title seems generic and a duplicate, but i've tried many of the options from previous questions, and I can't use a struct here
My system is using the messaging service NATS to sends maps between a subscriber and a publisher. The subscriber takes the received map, and inserts it as a document in a MongoDB collection
The problem I have is that floats and ints are inserted as strings!
In my code, the recipe is a configuration file that sets the datatypes of the columns received in the map. Think of it as a series of keys like this:
String column: "string",
Int column: "int"
Here's the code that creates the map with the right datatypes
mapWithCorrectDataTypes := make(map[string]interface{})
for columnNameFromDataTypesInRecipe, datatypeForColumnInRecipe := range dataTypesFromRecipeForColumns {
for natsMessageColumn, natsMessageColumnValue := range mapFromNATSMessage {
//If the column in the NATS message is found in the recipe, format the data as dictated in the recipe
if natsMessageColumn == columnNameFromDataTypesInRecipe {
if datatypeForColumnInRecipe.(string) == "string" {
natsMessageColumnValue = natsMessageColumnValue.(string)
mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe] = natsMessageColumnValue
}
if datatypeForColumnInRecipe.(string) == "int" {
convertedInt, err := strconv.Atoi(mapFromNATSMessage[columnNameFromDataTypesInRecipe].(string))
if err != nil {
fmt.Println("ERROR -->", err)
}
mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe] = convertedInt
}
if datatypeForColumnInRecipe.(string) == "float64" {
convertedFloat, err := strconv.ParseFloat(mapFromNATSMessage[columnNameFromDataTypesInRecipe].(string), 64)
if err != nil {
fmt.Println("ERROR -->", err)
}
mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe] = convertedFloat
fmt.Println("TYPE -->", reflect.TypeOf(mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe]))
}
} else {
//If column not found in the recipe, format as a string
mapWithCorrectDataTypes[natsMessageColumn] = natsMessageColumnValue.(string)
}
}
}
From the last line I put in a print statement for float64s to check that the datatype for this key in the map is correct, and it passes this test!
My question is this: If the data types are correctly being set in the map, why when the map is inserted as a document in MongoDB are the floats and ints set as strings?!
What I have tried so far:
Marshalling and unmarshalling the map as an interface, then inserting the record:
jsonVersionOfMap, err := json.Marshal(mapWithCorrectDataTypes)
if err != nil {
fmt.Println("ERROR -->", err)
}
var interfaceForJSON interface{}
json.Unmarshal(jsonVersionOfMap, &interfaceForJSON)
fmt.Println("JSON -->", interfaceForJSON)
err = mongoConnection.Insert(interfaceForJSON)
if err != nil {
fmt.Println("Error inserting MongoDB documents", err)
}
What am I missing here?
See the result with the incorrectly formatted data:
this may not be a fix. But i've resolved the issue i've been having. I'm using a publisher and a subscriber via NATS. Previously I was creating a map with all the data, then sending that out as a message, then the subscriber takes the map from the message, and processing the datatypes (on the subscriber side)
To fix the problem that I was experiencing, I instead formatted the maps' values on the publisher side. So I instead moved my code that checks for the datatype over to the NATS publisher, and not the code that processes the incoming message
I understand this isn't an ideal solution, but if you're using NATS and find you're having the same issue. Try this
I'm using the mgo driver for MongoDB, with the Gin framework.
type Users struct {
User_id *string `json:"id user" bson:"id user"`
Images []string `json:"images" bson:"images"`
}
I have this function which tries to convert the slice into JSON.
The slice here is UsersTotal
func GetUsersApi(c *gin.Context) {
UsersTotal, err := GetUsers()
if err != nil {
fmt.Println("error:", err)
}
c.JSON(http.StatusOK, gin.H{
"Count Users": len(UsersTotal),
"Users Found ": UsersTotal,
})
session, err := mgo.Dial(URL)
if err == nil {
fmt.Println("Connection to mongodb established ok!!")
cc := session.DB("UsersDB").C("results")
err22 := cc.Insert(&UsersTotal)
if err22 != nil {
fmt.Println("error insertion ", err22)
}
}
session.Close()
}
Running it I get the following error:
error insertion Wrong type for documents[0]. Expected a object, got a array.
Inserting multiple documents is the same as inserting a single one because the Collection.Insert() method has a variadic parameter:
func (c *Collection) Insert(docs ...interface{}) error
One thing you should note is that it expects interface{} values. Value of any type qualifies "to be" an interface{}. Another thing you should note is that only the slice type []interface{} qualifies to be []interface{}, a user slice []User does not. For details, see Type converting slices of interfaces in go
So simply create a copy of your users slice where the copy has a type of []interface{}, and that you can directly pass to Collection.Insert():
docs := make([]interface{}, len(UsersTotal))
for i, u := range UsersTotal {
docs[i] = u
}
err := cc.Insert(docs...)
// Handle error
Also please do not connect to MongodB in your handler. Do it once, on app startup, store the global connection / session, and clone / copy it when needed. For details see mgo - query performance seems consistently slow (500-650ms); and too many open files in mgo go server.
I am trying to connect to a mysql database.
I tried to see if I would get an error if I gave it wrong connection information but it still returns nil as error. Even If I shut down mysql completely it still does not return an error. What is the point of check for errors after this function if it does not return errors?
This is on Windows, I am using XAMPP and I don't have a password for the database. Username is "root".
import (
"database/sql"
"log"
_ "github.com/go-sql-driver/mysql"
)
func main() {
db, err := sql.Open("mysql", "root#tcp(127.0.0.1:3306)/dbname?charset=utf8")
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
SQL.Open only creates the DB object, but does not open any connections to the database. If you want to test your connections you have to execute a query to force opening a connection. The common way for this is to call Ping() on your DB object.
See http://golang.org/pkg/database/sql/#Open and http://golang.org/pkg/database/sql/#DB.Ping
Quoting from the doc of sql.Open():
Open may just validate its arguments without creating a connection to the database. To verify that the data source name is valid, call Ping.
As stated, Open() may not open any physical connection to the database server, but it will validate its arguments. That being said if arguments are valid, it may return nil error even if the database server is not reachable, or even if the host denoted by dataSourceName does not exist.
To answer your other question:
What is the point of check for errors after this function if it does not return errors?
You have to check returned errors because it can return errors. For example if the specified driverName is invalid, a non-nil error will be returned (see below).
To test if the database server is reachable, use DB.Ping(). But you can only use this if the returned error is nil, else the returned DB might also be nil (and thus calling the Ping() method on it may result in run-time panic):
if db, err := sql.Open("nonexistingdriver", "somesource"); err != nil {
fmt.Println("Error creating DB:", err)
fmt.Println("To verify, db is:", db)
} else {
err = db.Ping()
if err != nil {
fmt.Println("db.Ping failed:", err)
}
}
Output (try it on the Go Playground):
Error creating DB: sql: unknown driver "nonexistingdriver" (forgotten import?)
To verify, db is: <nil>
It turns out that this does not actually connect to the database right away.
Source: http://go-database-sql.org/accessing.html
To check the validity of connection, we can do this:
db, err := sql.Open("mysql", "root#tcp(127.0.0.1:3306)/dbname?charset=utf8")
err = db.Ping() // Need to do this to check that the connection is valid
if err != nil {
log.Fatal(err)
}
I'm using the github.com/go-sql-driver/mysql driver for go.
I open a database:
db, err := sql.Open("mysql", str)
Then I have two functions that are called 200 times each with following mysql code:
rows, err := db.Query("select name from beehives")
if err != nil {
panic(err)
}
defer rows.Close()
The second:
err = db.QueryRow("select id, secret, shortname from beehives where shortname = ?", beehive).Scan(&id, &secre
switch {
case err == sql.ErrNoRows:
err = errors.New("Beehive '"+beehive+"' not found.")
case err != nil:
panic("loginBeehive: "+ err.Error())
default:
// ... do the work
The first one is panicing.
How can there be more than one connection when I open the database only once and how do I close them?
sql.Open doesn't really open a connection to your database.
A sql.DB maintains a pool of connections to your database. Each time you query your database your program will try to get a connection from this pool or create a new one otherwise. These connections are than put back into the pool once you close them.
This is what rows.Close() does.
Your db.QueryRow("...") does the same thing internally when you call Scan(...).
The basic problem is that you're creating too many queries, of which each one needs a connection, but you are not closing your connections fast enough. This way your program has to create a new connection for each query.
You can limit the maximum number of connections your program uses by calling SetMaxOpenConns on your sql.DB.
See http://go-database-sql.org/surprises.html for more information.
The *DB object that you get back from sql.Open doesn't corresponds to a single connection. It is better thought as a handle for the database: it manages a connection pool for you.
You can control the number of open connections with `(*DB).SetMaxOpenConns and its pair for idling connections.
So basically what happens here is that db.Query and db.QueryRow tries to acquire a connection for themselves and the DB handle doesn't put any restrictions on the number of simultaneous connections so your code panics when it opens more than what mysql can handle.
Try to make prepared statements db.Prepare(query string) (*Stmt, error) and than stmt.Query or stmt.Exec and than stmt.Close to reuse connections.
hi can you try close connection after used
db, err := sql.Open("mysql", str)
defer db.Close() // close after end scope
My program is connecting always to database. (Realtime Face Recognition for Attendance)
Therefore opening and closing database connection is worthless.
Therefore it's keep opens the database connection only initializing the program.
func GetAllFaces() interface{} {
OpenDatabaseConnection() ...
}
But access database later, increased the no of connection and crashed the program. But closing the rows object kept no of active connection at minimum. (for me 1)
func SaveAttendance(faceId int, date time.Time) error {
sqlQuery := fmt.Sprintf("SELECT ... "))
rows, err := DB.Query(sqlQuery) ...
err = rows.Close()
return err
}