Currently i am using namednode, mysql for metastore and hive CLI on one single node. And another nodes as datanode. mysql is running on master (namenode). It works fine when i try to get metadata(show tables) hive(on master). Now i was trying to get the same metadata on datanode. But couldn't successed. I also tried starting thrift service on master and then tried to connect still didn't successed.
After reading this on the Apache wiki it looks like you have to go through the thift service if you have the metastore service running on a remote server.
I would just add the hive.metastore.uris property to your hive-site.xml and call it a day.
Related
I have confluent platform on my local machine i am just trying to read the data from aws ec2 instance i have credentials like hostname, DB name, pwd etc. I am using JDBC source connector. connector config is
name=test
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:mysql://ab.bca.bdc.aaa:abcd/DB?user=abc&password=bca
table.whitelist=ppp
mode=incrementing
after running connect standalone got error like
Invalid value java.sql.SQLNonTransientConnectionException: Cannot load connection class because of underlying exception: com.mysql.cj.exceptions.WrongArgumentException: Malformed database URL, failed to parse the connection string near
According to the JDBC MySQL syntax, user and password go before the database address as colon separated, not after
user:password#host_or_host_sublist
Not sure if using RDS or your own EC2 database would change the syntax of that
If you want to capture all database events, though, Debezium (your old question) would be what you want. Using the JDBC connector won't capture deletes (or events created and deleted between polls) and puts unnecessary strain on your database
Last night AWS RDS had an "Internet Connectivity Issue" that was resolved a short time later. However, my app (which runs in .NET Core and connects to an RDS MySQL instance via Pomelo.EntityFrameworkCore.MySql) could never re-establish a connection to the database even though the MySQL server was back online. I tested connecting from my own local machine and it worked just fine. I then re-deployed the .NET Core app it everything started working again.
Is there something that I need to re-create (the db context perhaps), or is there something that is being cached that I need to flush to try to connect again? I connect via hostname, and my connection string looks something like this:
server=something.somewhere.us-east-2.rds.amazonaws.com;userid=XXXX;password=YYYYY;database=ZZZZ
Here is the Exception being thrown:
MySqlException: Unable to connect to any of the specified MySQL hosts.
at MySqlConnector.Core.ServerSession+<ConnectAsync>d__56.MoveNext (C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:239)
and here is how I create my db context in Startup.cs:
services.AddDbContext<BlayFapContext>(opt => opt.UseMySql(Settings.Instance.SQLConnectionString));
Any help would be greatly appreciated.
Giawa
Okay, we worked out what happened. Pomelo's MySQL wrapper had an issue as outlined in their git repo here: https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/issues/434
Basically, if a MySQL database is not available when the connection string is first used then it will be cached as invalid and will never work again. You can easily confirm this by launching a service with no MySQL connectivity, verify it doesn't work, then launch MySQL and confirm that the service still doesn't work. It can never establish a MySQL connection after the first connection string is found to be invalid.
They patched it shortly after the 2.0.1 release, but they haven't updated Nuget with a new version since then, despite the issue being found 6 months ago. So, the fix is to checkout their repository source code, and patch it ourselves. We found the fix here works just fine: https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/pull/456
So, why was the connection string retried? We already had a successful connection! It turns out that the internet connectivity issue with the Ohio data center was not limited to RDS, but also affected EC2. Our EC2 instance was rebooted as part of the fix, and the MySQL connection wasn't valid when it reboot due to the continued connectivity issues. The state of that connection was cached, and even though the MySQL server came back online our service was toast.
Giawa
I have managed to successfully connect to a docker instance running MySQL via the mysql command line utility. However, I am struggling to do so via DataGrip application from JetBrains.
My mysql cli command is as follows:
mysql -h127.0.0.1 --port=8181 -uroot
The connection string that is generated in DataGrip is:
jdbc:mysql://127.0.0.1:8181
The error I am getting from DataGrip interface is:
[08001] Could not create connection to database server. Attempted reconnect 3 times. Giving up.
Is there anything that need to change in DataGrip that I am missing. I have read that the connection has to also be done via TCP. Not sure how to check that DataGrip is doing that.
Update: I eventually found the problem was the docker container I was using. It seems DataGrip is not able to connect to version 8.0.1 (mysql:8.0.1) mysql docker container. I tested using version 5.7 of the container and could successfully connect.
I was having an issue connecting MySQL 8.0.3 using DataGrip. You need to download the JDBC driver from Oracle website. Select Developer Releases (Since this is an unstable version). The 8.0.8 version worked for me. Download and save in a project folder or something similar on your computer. You will gonna need it later.
Go to DataGrip: File > DataSources. Click on the + and select Driver:
Screenshot of the Step above
On the section Driver Files > Additional files click on the + and select the jar file you just downloaded
After that on the Class dropdown select com.mysql.jdbc.Driver
Mark Dialect as MySQL
On the section URL templates, put the Name as default and Template as jdbc:mysql://{host::localhost}?[:{port::3306}][/{database}?][\?<&,user={user},password={password},{:identifier}={:identifier}>]
Click on apply
Done!
Screenshot of MySQL 8.0.3 running on Datagrip
Add a new datasource using the new driver.
If you find the following error:
[01S00] The server time zone value 'PDT' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the serverTimezone configuration property) to use a more specifc time zone value if you want to utilize time zone support
You can do a temporary fix going to the Advanced tab and setting the serverTimezone variable for UTC as an example.
There are no *.jar-files anymore. Only *.deb-packages for my OS on the Oracle site https://dev.mysql.com/downloads/connector/j/5.1.html
So I selected driver Amazon Aurora MySQL for my MySQL 8.0.3 in docker.
It works!
SnappyData v0.5
My goal is to start a "spark-shell" from my SnappyData install's /bin directory and issue Scala commands against existing tables in my SnappyData store.
I am on the same host as my SnappyData store, locator, and lead (and yes, they are all running).
To do this, I am running this command as per the documentation here:
Connecting to a Cluster with spark-shell
~/snappydata/bin$ spark-shell --master local[*] --conf snappydata.store.locators=10.0.18.66:1527 --conf spark.ui.port=4041
I get this error trying to create a spark-shell to my store:
[TRACE 2016/08/12 15:21:55.183 UTC GFXD:error:FabricServiceAPI
tid=0x1] XJ040 error occurred while starting server :
java.sql.SQLException(XJ040): Failed to start datab
ase 'snappydata', see the cause for details.
java.sql.SQLException(XJ040): Failed to start database 'snappydata',
see the cause for details.
at com.pivotal.gemfirexd.internal.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:124)
at com.pivotal.gemfirexd.internal.impl.jdbc.Util.newEmbedSQLException(Util.java:110)
at com.pivotal.gemfirexd.internal.impl.jdbc.Util.newEmbedSQLException(Util.java:136)
at com.pivotal.gemfirexd.internal.impl.jdbc.Util.generateCsSQLException(Util.java:245)
at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:3380)
at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedConnection.(EmbedConnection.java:450)
at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedConnection30.(EmbedConnection30.java:94)
at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedConnection40.(EmbedConnection40.java:75)
at com.pivotal.gemfirexd.internal.jdbc.Driver40.getNewEmbedConnection(Driver40.java:95)
at com.pivotal.gemfirexd.internal.jdbc.InternalDriver.connect(InternalDriver.java:351)
at com.pivotal.gemfirexd.internal.jdbc.InternalDriver.connect(InternalDriver.java:219)
at com.pivotal.gemfirexd.internal.jdbc.InternalDriver.connect(InternalDriver.java:195)
at com.pivotal.gemfirexd.internal.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:141)
at com.pivotal.gemfirexd.internal.engine.fabricservice.FabricServiceImpl.startImpl(FabricServiceImpl.java:290)
at com.pivotal.gemfirexd.internal.engine.fabricservice.FabricServerImpl.start(FabricServerImpl.java:60)
at io.snappydata.impl.ServerImpl.start(ServerImpl.scala:32)
Caused by: com.gemstone.gemfire.GemFireConfigException: Unable to
contact a Locator service (timeout=5000ms). Operation either timed out
or Locator does not exist. Configured list of
locators is "[dev-snappydata-1(null):1527]".
at com.gemstone.gemfire.distributed.internal.membership.jgroup.GFJGBasicAdapter.getGemFireConfigException(GFJGBasicAdapter.java:533)
at com.gemstone.org.jgroups.protocols.TCPGOSSIP.sendGetMembersRequest(TCPGOSSIP.java:212)
at com.gemstone.org.jgroups.protocols.PingSender.run(PingSender.java:82)
at java.lang.Thread.run(Thread.java:745)
hmm! I assume you are trying the Spark-shell from your desktop and connecting to the cluster in AWS?
Not sure this is going to work because the local JVM launched by spark-shell will attempt to connect to the p2p cluster in Snappydata which is not likely to work.
Snappy-shell on the other hand merely uses the JDBC client to connect (and, hence will work).
And, you cannot use the locator client port (1527), anyway. See here
Can you try with snappydata.store.locators=10.0.18.66:10334 NOT 1527 as the port ? Unlikely this will work but worth a try.
Maybe there is a way to open up all ports and access to these nodes on AWS. Not recommended for production, though.
I am curious for other responses from the engg team.
Until then, you may have to start the spark-shell from within the network (AWS node).
total newbie and totally lost...
I'm trying to setup coldfusion(8) with a mysql(5.5) database on a local (apache 2.2) webserver running Windows XP.
Coldfusion and Apache are running, but I don't know how to fiddle this in with MySQL.
I'm doing the following:
- Import file1.sql and file2.sql into MySql workbench (5.2) by reverse engineering
- Save this new model as base.mwb into my Apache htdocs folder
- Open the Coldfusion Administrator Data Sources
- Add new Source "myName", Database "base", localhost, 3306, root, password
When I try to add I always get Unknown database 'base' error.
I also tried to make a connection in MySQL or set up a new server instance in MySQL workbench, but these also fail because of no database "base".
I guess I'm missing something (probably a database :-)... Thanks for some pointers or a dummy tutorial. I'm longing for something to show up in http://localhost...
I would have to guess that you are coming from a background using Access DBs. MySQL and ColdFusion do not work together by have a DB file in the CF application. CF connects to a running MySQL server somewhere on your network (even on the same machine) using a JDBC connection. It does not look at a DB file and use that. So you will need to install MySQL Server and then import the DB into that. Then use CF's datasources to connect with that.