I'm collecting information and metrics about RDS Instances from CloudWatch and RDS services.
I tried to get metrics and information using the following methods:
DescribeDBInstances
GetMetricData
Unfortunately, there is no information about the Total Memory of DB Instance.
I'm assuming you want to get the memory for the underlying EC2 that you run your RDS.
If you need the memory allocated to the DB, it'll depend on the DB engine type. For example, in PostgreSQL, you need to query wal_buffers.
To get the instance details for your RDS, you need to get the instance type from rds.DescribeDBInstances and then query for the instance type details from ec2.DescribeInstanceTypes
import (
"context"
"log"
"strings"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/ec2"
ec2types "github.com/aws/aws-sdk-go-v2/service/ec2/types"
"github.com/aws/aws-sdk-go-v2/service/rds"
)
func getRdsMemoryMB(cfg aws.Config) int64 {
rdsClient := rds.NewFromConfig(cfg)
// get the instance details
input := rds.DescribeDBInstancesInput{
// if you omit this paramter, DescribeDBInstances will return the info for all instances
DBInstanceIdentifier: aws.String("instance-1-ARN"),
}
output, err := rdsClient.DescribeDBInstances(context.Background(),
&input,
func(opt *rds.Options) { opt.Region = "us-east-1" })
if err != nil {
log.Fatal(err)
}
// get the instance type details
ec2Client := ec2.NewFromConfig(cfg)
instanceName := strings.TrimPrefix(*output.DBInstances[0].DBInstanceClass, "db.")
params := ec2.DescribeInstanceTypesInput{
InstanceTypes: []ec2types.InstanceType{ec2types.InstanceType(instanceName)},
}
ec2Output, err := ec2Client.DescribeInstanceTypes(context.Background(), ¶ms)
if err != nil {
log.Fatal(err)
}
return *ec2Output.InstanceTypes[0].MemoryInfo.SizeInMiB
}
Related
After connecting to a Planetscale database using DSN with the following structure:
user:password#tcp(host)/database?tls=true
I decided to test it by running a simple query. Here's my main.go file:
package main
import (
"database/sql"
"fmt"
"os"
_ "github.com/go-sql-driver/mysql"
)
func GetDatabase() (*sql.DB, error) {
db, err := sql.Open("mysql", os.Getenv("DSN"))
return db, err
}
func main() {
db, err := GetDatabase()
if err != nil {
panic(err)
}
if err := db.Ping(); err != nil {
panic(err)
}
// ---
query, err := db.Query("SELECT name FROM status;")
if err != nil {
panic(err.Error())
}
var name string
for query.Next() {
err = query.Scan(&name)
if err != nil {
panic(err.Error())
}
fmt.Println(name)
}
// ---
fmt.Println("Successfully connected to PlanetScale!")
}
If I remove the section between the two comment lines, it will print out the 'successfully connected' message; however, if I try to run it including the query part, the outcome changes to:
panic: Error 1046: No database selected
Well, I decided to try with the query:
"USE database; SELECT name FROM status;"
However, the default driver won't take multiple statements and returns syntax error.
I know it seems like a simple question, but I already read the documentation from planetscale, from the go mysql driver, and I haven't been able to overcome this little issue. Any help will be greatly appreciated.
Looks like the problem is with the .env file. If I save the same information to a string variable within the same GetDatabase() function, and then pass it as argument instead of os.Getenv(), it connects normally. Can't tell exactly why that happens, but if someone can provide further information, I'll be thankful.
Edit:
when saving my credentials to an .env file, and using the built-in function os.Getenv("DSN"), the program will indeed connect to planetscale, but it won't select the given database in the DSN string. As mentioned in the question, selecting the database first and then running statements / queries doesn't work because of the constraints of the driver.
If I hard code the variables to a string variable, eg:
dsn := "root:mlc#tcp(127.0.0.1:3306)/app"
and then use that to open the connection, this time the program will use the desired database with any problem.
I'm using the mgo driver for MongoDB, with the Gin framework.
type Users struct {
User_id *string `json:"id user" bson:"id user"`
Images []string `json:"images" bson:"images"`
}
I have this function which tries to convert the slice into JSON.
The slice here is UsersTotal
func GetUsersApi(c *gin.Context) {
UsersTotal, err := GetUsers()
if err != nil {
fmt.Println("error:", err)
}
c.JSON(http.StatusOK, gin.H{
"Count Users": len(UsersTotal),
"Users Found ": UsersTotal,
})
session, err := mgo.Dial(URL)
if err == nil {
fmt.Println("Connection to mongodb established ok!!")
cc := session.DB("UsersDB").C("results")
err22 := cc.Insert(&UsersTotal)
if err22 != nil {
fmt.Println("error insertion ", err22)
}
}
session.Close()
}
Running it I get the following error:
error insertion Wrong type for documents[0]. Expected a object, got a array.
Inserting multiple documents is the same as inserting a single one because the Collection.Insert() method has a variadic parameter:
func (c *Collection) Insert(docs ...interface{}) error
One thing you should note is that it expects interface{} values. Value of any type qualifies "to be" an interface{}. Another thing you should note is that only the slice type []interface{} qualifies to be []interface{}, a user slice []User does not. For details, see Type converting slices of interfaces in go
So simply create a copy of your users slice where the copy has a type of []interface{}, and that you can directly pass to Collection.Insert():
docs := make([]interface{}, len(UsersTotal))
for i, u := range UsersTotal {
docs[i] = u
}
err := cc.Insert(docs...)
// Handle error
Also please do not connect to MongodB in your handler. Do it once, on app startup, store the global connection / session, and clone / copy it when needed. For details see mgo - query performance seems consistently slow (500-650ms); and too many open files in mgo go server.
github.com/ory/ladon is a library to manage role based access, written in golang. It contains a manager that is supposed to persist policies in the database and work with the database. The manager works fine for the in-memory case. When I use the manager to interface with sql, the required tables are not getting created.
db, err := sqlx.Open("mysql", "tx81:#tcp(127.0.0.1:3306)/policies")
......
err=db.Ping()
if err == nil {
fmt.Printf("Database is up")
}
warden := ladon.Ladon{
Manager: manager.NewSQLManager(db, nil),
}
var pol = &ladon.DefaultPolicy{
......
}
err = warden.Manager.Create(pol)
fmt.Printf("%s", err)
The error is printed as:
Table 'policies.ladon_policy' doesn't exist.
Why aren't the tables getting created?
Resources: https://github.com/ory/ladon#persistence
You need to call manager.CreateSchemas, which is never done by the manager itself.
You provide it with the schema name (postgresql only I believe) and table name to keep track of migration info.
eg:
if num, err := m.CreateSchemas("", "my_migration_table"); err != nil {
panic(err)
} else {
log.Infof("ran %d migrations", num)
}
It could probably use more documentation, you may want to file an issue with the author.
I am trying to connect to a mysql database.
I tried to see if I would get an error if I gave it wrong connection information but it still returns nil as error. Even If I shut down mysql completely it still does not return an error. What is the point of check for errors after this function if it does not return errors?
This is on Windows, I am using XAMPP and I don't have a password for the database. Username is "root".
import (
"database/sql"
"log"
_ "github.com/go-sql-driver/mysql"
)
func main() {
db, err := sql.Open("mysql", "root#tcp(127.0.0.1:3306)/dbname?charset=utf8")
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
SQL.Open only creates the DB object, but does not open any connections to the database. If you want to test your connections you have to execute a query to force opening a connection. The common way for this is to call Ping() on your DB object.
See http://golang.org/pkg/database/sql/#Open and http://golang.org/pkg/database/sql/#DB.Ping
Quoting from the doc of sql.Open():
Open may just validate its arguments without creating a connection to the database. To verify that the data source name is valid, call Ping.
As stated, Open() may not open any physical connection to the database server, but it will validate its arguments. That being said if arguments are valid, it may return nil error even if the database server is not reachable, or even if the host denoted by dataSourceName does not exist.
To answer your other question:
What is the point of check for errors after this function if it does not return errors?
You have to check returned errors because it can return errors. For example if the specified driverName is invalid, a non-nil error will be returned (see below).
To test if the database server is reachable, use DB.Ping(). But you can only use this if the returned error is nil, else the returned DB might also be nil (and thus calling the Ping() method on it may result in run-time panic):
if db, err := sql.Open("nonexistingdriver", "somesource"); err != nil {
fmt.Println("Error creating DB:", err)
fmt.Println("To verify, db is:", db)
} else {
err = db.Ping()
if err != nil {
fmt.Println("db.Ping failed:", err)
}
}
Output (try it on the Go Playground):
Error creating DB: sql: unknown driver "nonexistingdriver" (forgotten import?)
To verify, db is: <nil>
It turns out that this does not actually connect to the database right away.
Source: http://go-database-sql.org/accessing.html
To check the validity of connection, we can do this:
db, err := sql.Open("mysql", "root#tcp(127.0.0.1:3306)/dbname?charset=utf8")
err = db.Ping() // Need to do this to check that the connection is valid
if err != nil {
log.Fatal(err)
}
I'm using the github.com/go-sql-driver/mysql driver for go.
I open a database:
db, err := sql.Open("mysql", str)
Then I have two functions that are called 200 times each with following mysql code:
rows, err := db.Query("select name from beehives")
if err != nil {
panic(err)
}
defer rows.Close()
The second:
err = db.QueryRow("select id, secret, shortname from beehives where shortname = ?", beehive).Scan(&id, &secre
switch {
case err == sql.ErrNoRows:
err = errors.New("Beehive '"+beehive+"' not found.")
case err != nil:
panic("loginBeehive: "+ err.Error())
default:
// ... do the work
The first one is panicing.
How can there be more than one connection when I open the database only once and how do I close them?
sql.Open doesn't really open a connection to your database.
A sql.DB maintains a pool of connections to your database. Each time you query your database your program will try to get a connection from this pool or create a new one otherwise. These connections are than put back into the pool once you close them.
This is what rows.Close() does.
Your db.QueryRow("...") does the same thing internally when you call Scan(...).
The basic problem is that you're creating too many queries, of which each one needs a connection, but you are not closing your connections fast enough. This way your program has to create a new connection for each query.
You can limit the maximum number of connections your program uses by calling SetMaxOpenConns on your sql.DB.
See http://go-database-sql.org/surprises.html for more information.
The *DB object that you get back from sql.Open doesn't corresponds to a single connection. It is better thought as a handle for the database: it manages a connection pool for you.
You can control the number of open connections with `(*DB).SetMaxOpenConns and its pair for idling connections.
So basically what happens here is that db.Query and db.QueryRow tries to acquire a connection for themselves and the DB handle doesn't put any restrictions on the number of simultaneous connections so your code panics when it opens more than what mysql can handle.
Try to make prepared statements db.Prepare(query string) (*Stmt, error) and than stmt.Query or stmt.Exec and than stmt.Close to reuse connections.
hi can you try close connection after used
db, err := sql.Open("mysql", str)
defer db.Close() // close after end scope
My program is connecting always to database. (Realtime Face Recognition for Attendance)
Therefore opening and closing database connection is worthless.
Therefore it's keep opens the database connection only initializing the program.
func GetAllFaces() interface{} {
OpenDatabaseConnection() ...
}
But access database later, increased the no of connection and crashed the program. But closing the rows object kept no of active connection at minimum. (for me 1)
func SaveAttendance(faceId int, date time.Time) error {
sqlQuery := fmt.Sprintf("SELECT ... "))
rows, err := DB.Query(sqlQuery) ...
err = rows.Close()
return err
}