I am trying to connect to Mysql. I have uploaded the corresponding java jar we are using , which is mysql-connector-java-5.1.49.jar , I uploaded to s3 bucket. I am using the following code to access to Mysql and I am failing with the Error
An error occurred while calling o93.getDynamicFrame.
com.mysql.cj.jdbc.Driver
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext, SparkConf
from awsglue.context import GlueContext
from awsglue.job import Job
import time
from pyspark.sql.types import StructType, StructField, IntegerType, StringType
import boto3
import json
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
spark.conf.set("jars", "s3://xxxxx/jdbc-drivers/mysql-connector-java-5.1.49.jar")
client = boto3.client("secretsmanager" , region_name = "eu-west-1")
get_secret_value_response = client.get_secret_value(SecretId = "etl-1")
secret = get_secret_value_response["SecretString"]
secret=json.loads(secret)
username = secret.get("mysql_username")
password = secret.get("mysql_password")
url = secret.get("mysql_url")
table = secret.get("mysql_table")
connection_mysql_options_source_session = {
"url": url,
"dbtable": table,
"user": username,
"password": password,
"customJdbcDriverS3Path": "s3://xxxxx/jdbc-drivers/mysql-connector-java-5.1.49.jar",
"customJdbcDriverClassName": "com.mysql.cj.jdbc.Driver"}
# Read from JDBC databases with custom driver
df_session = glueContext.create_dynamic_frame.from_options(connection_type="mysql", connection_options=connection_mysql_options_source_session)
df_session.printSchema()
In job details section, I have referred to jar libs:
And I didn't define any connection in the connection section of job properties. I can't figure out why I am getting this error.
The strange thing is that I can connect with Crawler, data catalogue and also direct connection to the same server, but via Notebook & script I can't.
Related
I want to pass a SQL query as parameter in a POST request. Ideally this SQL query will be further passed into MySQL connection, and then it can fetch data back.
Here is what I did:
These are basic modules and settings:
from flask import Flask
from flask_cors import CORS, cross_origin
app = Flask(__name__)
CORS(app)
from app import app
from flaskext.mysql import MySQL
mysql = MySQL()
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = 'root'
app.config['MYSQL_DATABASE_DB'] = 'hibernate1'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
mysql.init_app(app)
This is the main.py, where I want to pass a SQL query:
from urllib import response
import pymysql
from app import app
from config import mysql
from flask import jsonify
from flask import flash, request
#app.route('/SQL/<query>', methods=['POST'])
def return_query(query):
conn = mysql.connect()
cursor = conn.cursor(pymysql.cursors.DictCursor)
cursor.execute(query)
Rows = cursor.fetchall()
respone = jsonify(Rows)
return respone
if __name__ == "__main__":
app.run()
This is test.py:
import requests
dictToSend = {"query": "select * from student"}
res = requests.post('http://localhost:5000/SQL', json = dictToSend)
print ('response from server:', res.text)
dictFromServer = res.json()
print(dictFromServer)
However, I got the following errors:
response from server: <!doctype html>
<html lang=en>
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
It says the requested URL was not found. But the following GET request works fine:
#app.route('/', methods=['GET'])
def return_hello():
return {"data": "hello"}
Could anyone tell me what I missed in the POST request? Thanks.
So I'm trying to deploy my Django project using lambda, with zappa. I'm using MySQL for DB engine. Now after doing some research, I realized that I needed to create a custom Django command to create DB, since I'm using MySQL. So I created crate_db command, zappa updated, then ran zappa manage dev create_db. Then I got this error: 2004 (HY000): Can't create TCP/IP socket (97)
below is my create_db.py file, for your information.
import sys
import logging
import mysql.connector
import os
from django.core.management.base import BaseCommand, CommandError
from django.conf import settings
rds_host = os.environ.get("MY HOST")
db_name = os.environ.get("")
user_name = os.environ.get("MY USERNAME")
password = os.environ.get("MY PASSWORD")
port = os.environ.get("3306")
logger = logging.getLogger()
logger.setLevel(logging.INFO)
class Command(BaseCommand):
help = 'Creates the initial database'
def handle(self, *args, **options):
print('Starting db creation')
try:
db = mysql.connector.connect(host=rds_host, user=user_name,
password=password, db="mysql", connect_timeout=10)
c = db.cursor()
print("connected to db server")
c.execute("""CREATE DATABASE bookcake_db;""")
c.execute("""GRANT ALL ON bookcake_db.* TO 'ryan'#'%'""")
c.close()
print("closed db connection")
except mysql.connector.Error as err:
logger.error("Something went wrong: {}".format(err))
sys.exit()
Any ideas? Thanks.
I have a Python Flask Server setup in an Ubuntu Machine and a MySQL from XAMPP as backend for the same.
How ever when I try to access the database tables from my python program it shows as
pymysql.err.InternalError: (1109, "Unknown table 'ALL_PLUGINS' in information_schema")
but i can access the database directly in MySQL admin page
the sample program I used to access the data.
from flaskext.mysql import MySQL
from flask import (Flask, request, session, g, redirect, url_for, abort, render_template, flash, Response)
import os
from werkzeug.utils import secure_filename
mysql = MySQL()
app = Flask(__name__)
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = 'root'
app.config['MYSQL_DATABASE_DB'] = 'information_schema'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
mysql.init_app(app)
#app.route('/')
def insert_student():
qry = "SELECT * FROM ALL_PLUGINS "
conn = mysql.connect()
cursor = conn.cursor()
cursor.execute(qry)
data = cursor.fetchall()
print(data)
conn.commit()
return "Sucess"
if __name__ == '__main__':
app.secret_key = 'super secret key'
app.debug = True
app.run()
instead of normal running I ge the following
the screen shot
Mysql does not have an all_plugins table in information schema. The plugins table (well, view) is called plugins.
So, your query should be:
SELECT * FROM PLUGINS
Based on the comment from #snakecharmerb:
Mariadb, on the other hand, does have all_plugins table, which presumably is the cause of the confusion.
I am working on a Spark Application and I want to create a rest API in Django, below is my code
from django.shortcuts import render
from django.http import Http404
from rest_framework.views import APIView
from rest_framework.decorators import api_view
from rest_framework.response import Response
from rest_framework import status
from django.http import JsonResponse
from django.core import serializers
from django.conf import settings
import json
from pyspark import SparkContext, SparkConf, SQLContext
sc = SparkContext()
sql = SQLContext(sc)
df = Sql.read.format("jdbc").options(
url = "jdbc:mysql://127.0.0.1:3306/demo",
driver = "com.mysql.cj.jdbc.Driver",
dbtable = "tablename",
user = "xyz",
password = "abc"
).load()
totalrecords = df.count()
# Create your views here.
#api_view(["GET"])
def Demo(self):
try:
a = str(totalrecords)
return JsonResponse(a,safe=False)
except ValueError as e:
return Response(e.args[0],status.HTTP_400_BAD_REQUEST)
I want to know how will I run this code, as I have directly tried "python manage.py runserver" which is not working, so how to run this spark and django with django api and spark-submit with all required spark jar file?
To run this code you have to use spark submit only,
spark-submit --jars mysql.jar manage.py runserver 0.0.0.0:8000
or
spark-submit manage.py runserver
I am reading the tutorial here
But this tutorial uses sqlalchemy as following:
from datetime import datetime, timedelta
import unittest
from app import app, db
from app.models import User, Post
class UserModelCase(unittest.TestCase):
def setUp(self):
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite://'
db.create_all()
def tearDown(self):
db.session.remove()
db.drop_all()
in fact, I already wrote a web app that uses from flask_mysqldb import MySQL
the __init__.py in the tutorial looks like this:
from flask import Flask
from config import Config
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
app = Flask(__name__)
app.config.from_object(Config)
db = SQLAlchemy(app)
migrate = Migrate(app, db)
from app import routes, models
I have mysql=MySQL(app) in my own __init__.py
can I replace db by mysql in the first part of the code to test my app?
and what about app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite://'?
I haven't use the sqlalchemy,just use flask_mysqldb
Sol:
use testing.mysqld module
here
Reference:
Mock a MySQL database in Python