I read in the docs that in order to use Google Direction API Web Service, I needed to use IP restrictions. So to achieve this feature, I've created a proxy server that serve me the direction I want. The link looks like so:
https://myapp-44th7.uc.r.appspot.com/?origin=22.174181,33.6436763&destination=22.189821,33.640532
When I use an unrestricted API key and I access this URL in a browser, it works fine, I get the desired JSON file. The problem comes when I want to restrict it. I cannot add a single IP address as a restriction because my Android app is used by users with many different IP addresses. I have hard times trying understanding how to restrict this. Please help solve this issue. What IP should I add?
Edit:
package main
import (
"fmt"
"net/http"
"io/ioutil"
"context"
"google.golang.org/appengine"
"google.golang.org/appengine/urlfetch"
)
const directionAPIKey = "..............uFvjU"
const directionURL = "https://maps.googleapis.com/maps/api/directions/json?origin=%s&destination=%s&mode=%s&key=%s"
func main() {
http.HandleFunc("/", handler)
appengine.Main()
}
func handler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
direction, err := fetchDirection(ctx, r.FormValue("origin"), r.FormValue("destination"), r.FormValue("mode"))
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Add("Content-Type", "application/json; charset=utf-8")
w.Write(direction)
}
func fetchDirection(ctx context.Context, origin string, destination string, mode string) ([]byte, error) {
client := urlfetch.Client(ctx)
resp, err := client.Get(fmt.Sprintf(directionURL, origin, destination, mode, directionAPIKey))
if err != nil {
return nil, err
}
defer resp.Body.Close()
return ioutil.ReadAll(resp.Body)
}
The directionAPIKey being the key that I use in my URL to access:
https://myapp-44th7.uc.r.appspot.com/?origin=22.174181,33.6436763&destination=22.189821,33.64053&key=..............uFvjU
And the one for which I added the restriction in the GCP. 172.217.23.52 being the IP that comes when I ping myapp-44th7.uc.r.appspot.com.
For you to restrict your API keys with IP restrictions, you need to add your server's public facing IP address as a restriction to your API key. More information here.
Once you have applied IP restrictions to your API key, only requests from your specific server(in your case your proxy server) is allowed to use your restricted API key. This means that all requests which come from different IP addresses other than the one you have set as restriction are blocked. This is the reason why you are getting an error if you tried to send a request directly from your browser using your restricted API key since your browser has a different IP address than your proxy server.
To make sure that your restricted API key works, you can verify if you can successfully send a Directions API request through your proxy server using your restricted API key.
Hope this helps!
Related
This question already has an answer here:
Does ReadString() discard bytes following newline?
(1 answer)
Closed 8 months ago.
I'm trying to write a TCP client that sends out a JSON RPC to a locally hosted TCP server. The server should immediately return two replies.
The TCP server is hosted on port 60000.
This is what the client code looks like:
package main
import (
"fmt"
"log"
"bufio"
"net"
)
func main() {
d := net.Dialer{ }
c, err := d.Dial("tcp", "127.0.0.1:60000")
if err != nil {
log.Println(err)
return
}
data := `{"id": 1,"method":"mining.configure","params": [["version-rolling"],{"version-rolling.mask": "1fffe000","version-rolling.min-bit-count": 16}]}`
fmt.Fprintf(c, data+"\n")
for {
message, _ := bufio.NewReader(c).ReadString('\n')
fmt.Println(message)
}
}
This is what I'm sending out ( the "data" variable )
{
"id": 1,
"method":"mining.configure",
"params": [["version-rolling"],
{"version-rolling.mask": "1fffe000",
"version-rolling.min-bit-count": 16}]
}
This is the expected reply:
{"error":null,"id":1,"result":{"version-rolling":true,"version-rolling.mask":"1fffe000"}}
{"id":null,"method":"mining.set_version_mask","params":["1fffe000"]}
Most of the time I only get the first reponse (has the result "field") without the second JSON that has the "method" field. Sometimes I get both.
But this is the usual reply:
{"error":null,"id":1,"result":{"version-rolling":true,"version-rolling.mask":"1fffe000"}}
I know that when code becomes non-deterministic like this it's because of an asynchronous issue going on.
But every beginner tutorial I encountered teaches this exact structure for writing a TCP client. What's missing from it?
I've tested the same RPC call to the server extensively with Python with the Twisted framework and I get back the both replies 100% of the time. This makes me certain the issue is within my client code and not the server.
Depending on the timing that data is received, the call to ReadString can buffer data past the \n. The code in the question discards the reader and data buffered in the reader on each iteration of the loop.
To avoid discarding data, create the reader once and reuse inside the loop:
r := bufio.NewReader(c)
for {
message, _ := r.ReadString('\n')
fmt.Println(message)
}
TL;DR
I'm wondering how to organise function maps for go templates that are dependant on the data being obtained in the specific endpoint being called.
Issue
I have a main frontend endpoint (for a CMS) which will grab the slug of the page, and go and obtain the post data it from the model. I also have a templates package that takes in models, the post data and gin context.
GoView is currently being used. But I am setting HTMLRender to equal a new ginview evertime the page is hit presumably this isn't great for performance and I'm wondering if there is a neater way of going about it.
// Serve the front end website
func (c *FrontendController) Serve(g *gin.Context) {
path := g.Request.URL.Path
post, err := c.models.Posts.GetBySlug(path)
if err != nil {
NoPageFound(g)
return
}
tf := templates.NewFunctions(g, c.models, &post)
c.server.HTMLRender = ginview.New(goview.Config{
Root: paths.Theme(),
Extension: config.Template["file_extension"],
Master: "/layouts/main",
Partials: []string{},
DisableCache: true,
Funcs: tf.GetFunctions(),
})
r := ResourceData{
Post: post,
}
g.HTML(200, config.Template["template_dir"] + "/" + post.PageTemplate, r)
}
Templates
Some template functions include getting the field & author of the post, and seeing if the user is logged in, for example.
func (t *TemplateFunctions) isAuth() bool {
...
}
There are also template functions that aren't tied to gin.context such as getting the assets path.
func (t *TemplateFunctions) assetsPath() string {
...
}
Questions
Is there a way to merge the data dependant functions with global ones?
Is there a way to set the template functions without redeclaring the HTMLRender every time a page is hit? Or is it necessary?
I am trying to achieve something similar to the Go Playground's share button.
When a button called share is clicked, the current HTML response is saved into a file. That also includes everything a user can see.
What I can do so far
- I can successfully save an HTML page into a file.
- I can get and save a whole page from a URL using HTTP Get.
Caveats
- I can't save a current webpage I am working on.
This is my code so far for getting remote URLs:
func HTTPGet(url string, timeout time.Duration) (content []byte, err error) {
request, err := http.NewRequest("POST", url, nil)
if err != nil {
return
}
ctx, cancel_func := context.WithTimeout(context.Background(), timeout)
request = request.WithContext(ctx)
response, err := http.DefaultClient.Do(request)
if err != nil {
return
}
defer response.Body.Close()
if response.StatusCode != 200 {
cancel_func()
return nil, fmt.Errorf("INVALID RESPONSE; status: %s", response.Status)
}
return ioutil.ReadAll(response.Body)
}
Hopefully this will not involve messing with the DOM manually.
There is a few ways this can be done, with some cooperation from the client.
If you want to save the current web page as it is displayed to the client (with the current form values, etc.), you have to do that with Javascript on the client side. You can get the document, serialize it in some form and send it to the server to save it.
Alternative is to generate the HTML page on the server side again, but that will not contain data user entered, or any dynamically generated HTML on the client side.
Another way is to use Javascript on the client side to collect page state including form values and data representing any dynamically generated content, send it to the server to save, and reconstruct the page as needed.
I'm currently using MySQL Workbench on Windows 8.1 to access a remote MySQL database on a Linux server using Standard TCP/IP over SSH. Basically I have the following information:
SSH Hostname: dbserver.myorg.com:ssh-port
SSH Username: myRemoteLoginUsername
SSH Password: (stored in vault)
SSH Key File: path to a local .ppk file
MySQL Hostname: 127.0.0.1
MySQL Server Port: 3306
Username: myRemoteDbUsername
Password: (stored in vault)
Default schema: myRemoteDatabaseName
How can I connect to the database from a Go command application using github.com/go-sql-driver/mysql?
how should my DataSourceName string in the sql.Open statement look like?
db, err := sql.Open("mysql", <DataSourceName> ) {}
Is there any extra work needed to prepare a working DataSourceName string?
On my Windows PC I have putty installed. I read about tunneling and added a Dynamic tunnel for port 3306 (D3306). I expected this will let me connect using a connection to localhost:3306 and automatically forward the request to the remote db whenever I'm connected to the remote host with putty, but this didn't work as expected either.
I promised to provide my example, here it comes. Basically my solution establishes an ssh tunnel to the remote server and queries the remote database through this tunnel. The ssh tunnel is part of the solution.
The first thing I had to do is to convert my PuTTY .ppk private key file into a valid OpenSSH .pem key file. This can easily be done using the Export feature in PuTTYgen. As I wanted to support password encrypted private keys I also needed a function to decrypt the key and reformat it from its decrypted raw format into a valid format accepted by golang.org/x/crypto/ssh/ParsePrivateKey, which is needed to get the list of signers for authentication.
The solution itself consists of a package contained in two files. The main part of the application is done in main.go which contains all relevant data assignments as well as the code related to the database query. Everything related to ssh tunneling and key handling is contained in sshTunnel.go.
The solution does not provide a mechanismn for a secure password store, nor does it ask for a password. The password is provided in the code. However, it would not be too complicated to implement a callback method for password requests.
Please note: from a performance perspective this is not an ideal solution. It also lacks of proper error handling. I have provided this as an example.
The example is a tested and working example. I developed and used this from a Windows 8.1 PC. The database server is on a remote Linux system. All you need to change is the data and the query part in main.go.
Here is the first part contained in main.go:
// mysqlSSHtunnel project main.go
// Establish an ssh tunnel and connect to a remote mysql server using
// go-sql-driver for database queries. Encrypted private key pem files
// are supported.
//
// This is an example to give an idea. It's far from a performant solution. It
// lacks of proper error handling and I'm sure it could really be much better
// implemented. Please forgive me, as I just started with Go about 2 weeks ago.
//
// The database used in this example is from a real Opensimulator installation.
// It queries the migrations table in the opensim database.
//
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"os"
)
// Declare your connection data and user credentials here
const (
// ssh connection related data
sshServerHost = "test.example.com"
sshServerPort = 22
sshUserName = "tester"
sshPrivateKeyFile = "testkey.pem" // exported as OpenSSH key from .ppk
sshKeyPassphrase = "testoster0n" // key file encrytion password
// ssh tunneling related data
sshLocalHost = "localhost" // local localhost ip (client side)
sshLocalPort = 9000 // local port used to forward the connection
sshRemoteHost = "127.0.0.1" // remote local ip (server side)
sshRemotePort = 3306 // remote MySQL port
// MySQL access data
mySqlUsername = "opensim"
mySqlPassword = "h0tgrits"
mySqlDatabase = "opensimdb"
)
// The main entry point of the application
func main() {
fmt.Println("-> mysqlSSHtunnel")
tunnel := sshTunnel() // Initialize sshTunnel
go tunnel.Start() // Start the sshTunnel
// Declare the dsn (aka database connection string)
// dsn := "opensim:h0tgrits#tcp(localhost:9000)/opensimdb"
dsn := fmt.Sprintf("%s:%s#tcp(%s:%d)/%s",
mySqlUsername, mySqlPassword, sshLocalHost, sshLocalPort, mySqlDatabase)
// Open the database
db, err := sql.Open("mysql", dsn)
if err != nil {
dbErrorHandler(err)
}
defer db.Close() // keep it open until we are finished
// Simple select query to check migrations (provided here as an example)
rows, err := db.Query("SELECT * FROM migrations")
if err != nil {
dbErrorHandler(err)
}
defer rows.Close()
// Iterate though the rows returned and print them
for rows.Next() {
var version int
var name string
if err := rows.Scan(&name, &version); err != nil {
dbErrorHandler(err)
}
fmt.Printf("%s, %d\n", name, version)
}
if err := rows.Err(); err != nil {
dbErrorHandler(err)
}
// Done for now
fmt.Println("<- mysqlSSHtunnel")
}
// Simple mySql error handling (yet to implement)
func dbErrorHandler(err error) {
switch err := err.(type) {
default:
fmt.Printf("Error %s\n", err)
os.Exit(-1)
}
}
Now the second part in sshTunnel.go:
// mysqlSSHtunnel project sshTunnel.go
//
// Everything regarding the ssh tunnel goes here. Credits go to Svett Ralchev.
// Look at http://blog.ralch.com/tutorial/golang-ssh-tunneling for an excellent
// explanation and most ssh-tunneling related details used in this code.
//
// PEM key decryption is valid for password proected SSH-2 RSA Keys generated as
// .ppk files for putty and exported as OpenSSH .pem keyfile using PuTTYgen.
//
package main
import (
"bytes"
"crypto/x509"
"encoding/base64"
"encoding/pem"
"fmt"
"golang.org/x/crypto/ssh"
"io"
"io/ioutil"
"net"
)
// Define an endpoint with ip and port
type Endpoint struct {
Host string
Port int
}
// Returns an endpoint as ip:port formatted string
func (endpoint *Endpoint) String() string {
return fmt.Sprintf("%s:%d", endpoint.Host, endpoint.Port)
}
// Define the endpoints along the tunnel
type SSHtunnel struct {
Local *Endpoint
Server *Endpoint
Remote *Endpoint
Config *ssh.ClientConfig
}
// Start the tunnel
func (tunnel *SSHtunnel) Start() error {
listener, err := net.Listen("tcp", tunnel.Local.String())
if err != nil {
return err
}
defer listener.Close()
for {
conn, err := listener.Accept()
if err != nil {
return err
}
go tunnel.forward(conn)
}
}
// Port forwarding
func (tunnel *SSHtunnel) forward(localConn net.Conn) {
// Establish connection to the intermediate server
serverConn, err := ssh.Dial("tcp", tunnel.Server.String(), tunnel.Config)
if err != nil {
fmt.Printf("Server dial error: %s\n", err)
return
}
// access the target server
remoteConn, err := serverConn.Dial("tcp", tunnel.Remote.String())
if err != nil {
fmt.Printf("Remote dial error: %s\n", err)
return
}
// Transfer the data between and the remote server
copyConn := func(writer, reader net.Conn) {
_, err := io.Copy(writer, reader)
if err != nil {
fmt.Printf("io.Copy error: %s", err)
}
}
go copyConn(localConn, remoteConn)
go copyConn(remoteConn, localConn)
}
// Decrypt encrypted PEM key data with a passphrase and embed it to key prefix
// and postfix header data to make it valid for further private key parsing.
func DecryptPEMkey(buffer []byte, passphrase string) []byte {
block, _ := pem.Decode(buffer)
der, err := x509.DecryptPEMBlock(block, []byte(passphrase))
if err != nil {
fmt.Println("decrypt failed: ", err)
}
encoded := base64.StdEncoding.EncodeToString(der)
encoded = "-----BEGIN RSA PRIVATE KEY-----\n" + encoded +
"\n-----END RSA PRIVATE KEY-----\n"
return []byte(encoded)
}
// Get the signers from the OpenSSH key file (.pem) and return them for use in
// the Authentication method. Decrypt encrypted key data with the passphrase.
func PublicKeyFile(file string, passphrase string) ssh.AuthMethod {
buffer, err := ioutil.ReadFile(file)
if err != nil {
return nil
}
if bytes.Contains(buffer, []byte("ENCRYPTED")) {
// Decrypt the key with the passphrase if it has been encrypted
buffer = DecryptPEMkey(buffer, passphrase)
}
// Get the signers from the key
signers, err := ssh.ParsePrivateKey(buffer)
if err != nil {
return nil
}
return ssh.PublicKeys(signers)
}
// Define the ssh tunnel using its endpoint and config data
func sshTunnel() *SSHtunnel {
localEndpoint := &Endpoint{
Host: sshLocalHost,
Port: sshLocalPort,
}
serverEndpoint := &Endpoint{
Host: sshServerHost,
Port: sshServerPort,
}
remoteEndpoint := &Endpoint{
Host: sshRemoteHost,
Port: sshRemotePort,
}
sshConfig := &ssh.ClientConfig{
User: sshUserName,
Auth: []ssh.AuthMethod{
PublicKeyFile(sshPrivateKeyFile, sshKeyPassphrase)},
}
return &SSHtunnel{
Config: sshConfig,
Local: localEndpoint,
Server: serverEndpoint,
Remote: remoteEndpoint,
}
}
Well, you could do that "full Go", I think.
The SSH part and port-forwarding
I'd start with something like this (I failed to google a better example).
Note two problems with this code:
It's not actually correct: it connects to a remote socket
before accepting a client connection
while it should do the reverse: accept a client connection to
a port-forwarded local socket then use the active SSH session to connect
to the remote socket and if this succeeds, spawn two goroutines to shovel
the data between those two sockets.
When configuring the SSH client, it explicitly allows password-based
authentication for unknown reason. You don't need this as you're using
pubkey-based auth.
An obstacle which might trip you there is managing an access to your SSH key.
The problem with it is that a good key should be protected by a passphrase.
You say the key's password is "stored in valut" and I honestly have no idea what "the valut" is.
On the systems I use, an SSH client either asks for the password to decrypt the key or work with a so-called "SSH agent":
On Linux-based systems it's typically an OpenSSH's ssh-agent binary working in the background which is accessed via a Unix-domain socket, and located by inspecting an environment variable named SSH_AUTH_SOCK.
On Windows I use PuTTY, and it has its own agent, pageant.exe.
I'm don't know which means PuTTY SSH client uses to locate it.
To access the OpenSSH agent, golang.org/x/crypto/ssh provides the agent subpackage which can be used to locate the agent and communicate with it.
If you need to obtain keys from pageant, I'm afraid you'll need to figure out what protocol that uses and implement it.
The MySQL part
The next step would be to integrate this with go-sql-driver.
I'd start the simplest way possible:
When you have your SSH port forwarding working,
make it listen for incoming connections on a random port on localhost.
When the connection is opened, get the port from the returned connection
object.
Use that port number to constuct the connection string to pass to an instance of sql.DB you will create to use go-sql-driver.
The driver will then connect to your port-forwarded port, and your SSH layer will do the rest.
After you have this working, I'd explore whether the driver of your choice
allows some more fine-grained tweaking like allowing you to directly pass it an instance of io.ReadWriter (an opened socket) so that you could skip the port-forwarding setup entirely and just produce new TCP connections forwarded through SSH, that is, skip the "listening locally" step.
I'm new to developing web applications in Go.
I'm looking for the best way to integrate a MySQL database into my web application.
I was thinking of doing something like this:
type Context struct {
Database *sql.DB
}
// Some database methods like Close() and Query() for Context struct here
In the main function for my web application I would have something like this:
db := sql.Open(...)
ctx := Context{db}
I would then pass then pass my Context structure into various handlers that require a database connection. Would this be a good design decision or is there a better way to integrate a SQL database into my web application?
I typically do something like this:
package main
func main(){
db,err := sql.Open(...)
if err != nil {
log.Fatal(err)
}
defer db.Close()
http.HandleFunc("/feed", server.FeedHandler(db))
http.HandleFunc("/gui", server.GuiHandler(db))
...
log.Fatal(http.ListenAndServe(":8000", nil))
}
Where server is a separate package where I define, implement and test all my http handlers.
This is basically what you were thinking of but skipping the step of wrapping the db in a struct which isn't necessary. I would not recommend making the db a global variable. Having global dependencies will completely destroy any possibility of testing your http handlers in a reliable fashion.
Dependency injecting the db, as in the example above, costs you two extra characters to type every time you call the function but it allows you to test your http handlers easily using the go-sqlmock package which you definitely want to do as your web app grows.
package server
func TestFeedHandler(t *testing.T){
mockdb, err := sqlmock.New()
if err != nil {
t.Errorf("An error '%s' was not expected when opening a stub database connection", err)
}
columns := []string{"id", "name"}
sqlmock.ExpectQuery("SELECT id,name from persons;").
WillReturnRows(sqlmock.NewRows(columns).
AddRow(1234, "bob")
handler := FeedHandler(mockdb)
// test that handler returns expected result
}