I am trying to do a SQL database in Google Cloud SQL and with a Java project make many queries to the database, but I have this answer:
Buckets:
Exception in thread "main" com.google.cloud.storage.StorageException: java.lang.NullPointerException: Required parameter project must be specified.
at com.google.cloud.storage.StorageException.translateAndThrow(StorageException.java:73)
at com.google.cloud.storage.StorageImpl.listBuckets(StorageImpl.java:288)
at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:257)
at package1.Auth.authExplicit(Auth.java:46)
at package1.Main.main(Main.java:21)
Caused by: java.lang.NullPointerException: Required parameter project must be specified
public static void main(String[] args) throws IOException {
String instanceConnectionName="";
String databaseName="prueba";
String username="root";
String password="root";
String sql="SELECT * FROM alumnos";
String jsonPath="/Users/franciscomanuelpozodelmoral/Downloads/Team5-c4e75c7cb498.json";
Auth.authExplicit(jsonPath);
try {
Class.forName("com.mysql.jdbc.Driver");
String jdbcURL=String.format("jdbc:mysql://google/%s?cloudSqlInstance=%s&" +
"socketFactory=com.google.cloud.sql.mysql.SocketFactory",databaseName,instanceConnectionName);
Connection connection=DriverManager.getConnection(jdbcURL, username, password);
Statement statement=connection.createStatement();
ResultSet rs=statement.executeQuery(sql);
while(rs.next()) {
System.out.println(rs.getString("DNI") + rs.getString("Nombre") + rs.getString("Apellidos") + rs.getInt("Telefono")+rs.getInt("Curso"));
}
} catch(Exception e) {
e.printStackTrace();
}
}
public class Auth {
static void authImplicit() {
// If you don't specify credentials when constructing the client, the client library will
// look for credentials via the environment variable GOOGLE_APPLICATION_CREDENTIALS.
//Storage storage = StorageOptions.getDefaultInstance().getService();
Storage storage=StorageOptions.getDefaultInstance().getService();
System.out.println("Buckets:");
Page<Bucket> buckets = storage.list();
for (Bucket bucket : buckets.iterateAll()) {
System.out.println(bucket.toString());
}
}
static void authExplicit(String jsonPath) throws IOException {
// You can specify a credential file by providing a path to GoogleCredentials.
// Otherwise credentials are read from the GOOGLE_APPLICATION_CREDENTIALS environment variable.
GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream(jsonPath))
.createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"));
Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();
System.out.println("Buckets:");
Page<Bucket> buckets = storage.list();
// for (Bucket bucket : buckets.iterateAll()) {
// System.out.println(bucket.toString());
// }
}
}
Can you help me? Thanks (The instanceConnectionName is group5...)
Related
I'm setting up CAS 5.3 to do Bind authentication agains an Oracle database. But I don't know how to setup a OracleDataSource while using WAR Overlay method. Any guidance would be appreciated, thanks.
Just setting up Driver and URL doesn't appear to work. It seems the HikariDataSource is used by default, and it doesn't implement the required getConnection(String username, String password).
#SneakyThrows
public static DataSource newDataSource(final AbstractJpaProperties jpaProperties) {
final String dataSourceName = jpaProperties.getDataSourceName();
final boolean proxyDataSource = jpaProperties.isDataSourceProxy();
if (StringUtils.isNotBlank(dataSourceName)) {
try {
final JndiDataSourceLookup dsLookup = new JndiDataSourceLookup();
dsLookup.setResourceRef(false);
final DataSource containerDataSource = dsLookup.getDataSource(dataSourceName);
if (!proxyDataSource) {
return containerDataSource;
}
return new DataSourceProxy(containerDataSource);
} catch (final DataSourceLookupFailureException e) {
LOGGER.warn("Lookup of datasource [{}] failed due to {} falling back to configuration via JPA properties.", dataSourceName, e.getMessage());
}
}
final HikariDataSource bean = new HikariDataSource();
if (StringUtils.isNotBlank(jpaProperties.getDriverClass())) {
bean.setDriverClassName(jpaProperties.getDriverClass());
}
bean.setJdbcUrl(jpaProperties.getUrl());
bean.setUsername(jpaProperties.getUser());
bean.setPassword(jpaProperties.getPassword());
bean.setLoginTimeout((int) Beans.newDuration(jpaProperties.getPool().getMaxWait()).getSeconds());
bean.setMaximumPoolSize(jpaProperties.getPool().getMaxSize());
bean.setMinimumIdle(jpaProperties.getPool().getMinSize());
bean.setIdleTimeout((int) Beans.newDuration(jpaProperties.getIdleTimeout()).toMillis());
bean.setLeakDetectionThreshold(jpaProperties.getLeakThreshold());
bean.setInitializationFailTimeout(jpaProperties.getFailFastTimeout());
bean.setIsolateInternalQueries(jpaProperties.isIsolateInternalQueries());
bean.setConnectionTestQuery(jpaProperties.getHealthQuery());
bean.setAllowPoolSuspension(jpaProperties.getPool().isSuspension());
bean.setAutoCommit(jpaProperties.isAutocommit());
bean.setValidationTimeout(jpaProperties.getPool().getTimeoutMillis());
return bean;
}
I would require that the bean created above, would be a OracleDataSource instance.
I have a Hadoop job running in EMR and i am passing the S3 Path as input and output to this Job.
When i run locally everything is working fine.( As there is a single node)
How ever when i run in EMR with 5 node cluster i am running into File Already exists IO Exception.
The output path has a timestamp in it so the out put path doesn't exists in S3.
Error: java.io.IOException: File already exists:s3://<mybucket_name>/8_9_0a4574ca-96d0-47c8-8eb8-4deb82944d4b/customer/RawFile12.txt/1523583593585/TOKENIZED/part-m-00000
I have a very simple hadoop app (primarily my mapper) which reads each line from a file and converts it (using an existing library)
Not sure why each node is trying to write with the same file name.
Here is mapper
public static class TokenizeMapper extends Mapper<Object,Text,Text,Text>{
public void map(Object key, Text value,Mapper.Context context) throws IOException,InterruptedException{
//TODO: Invoke Core Engine to transform the Data
Encryption encryption = new Encryption();
String tokenizedVal = encryption.apply(value.toString());
context.write(tokenizedVal,1);
}
}
Any my Reducer
public static class TokenizeReducer extends Reducer<Text,Text,Text,Text> {
public void reduce(Text text,Iterable<Text> lines,Context context) throws IOException,InterruptedException{
Iterator<Text> iterator = lines.iterator();
int counter =0;
while(iterator.hasNext()){
counter++;
}
Text output = new Text(""+counter);
context.write(text,output);
}
}
And my main class
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
long startTime = System.currentTimeMillis();
try {
Configuration config = new Configuration();
String[] additionalArgs = new GenericOptionsParser(config, args).getRemainingArgs();
if (additionalArgs.length != 2) {
System.err.println("Usage: Tokenizer Input_File and Output_File ");
System.exit(2);
}
Job job = Job.getInstance(config, "Raw File Tokenizer");
job.setJarByClass(Tokenizer.class);
job.setMapperClass(TokenizeMapper.class);
job.setReducerClass(TokenizeReducer.class);
job.setNumReduceTasks(0);
job.setOutputKeyClass(Text.class);
job.setOutputKeyClass(Text.class);
FileInputFormat.addInputPath(job, new Path(additionalArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(additionalArgs[1]));
boolean status = job.waitForCompletion(true);
if (status) {
//System.exit(0);
System.out.println("Completed Job Successfully");
} else {
System.out.println("Job did not Succeed");
}
}
catch(Exception e){
e.printStackTrace();
}
finally{
System.out.println("Total Time for processing =["+(System.currentTimeMillis()-startTime)+"]");
}
}
I am passing the arguments when i am launching the cluster as
s3://<mybucket>/8_9_0a4574ca-96d0-47c8-8eb8-4deb82944d4b/customer/RawFile12.txt
s3://<mybucket>/8_9_0a4574ca-96d0-47c8-8eb8-4deb82944d4b/customer/RawFile12.txt/1523583593585/TOKENIZED
Appreciate any inputs.
Thanks
In the driver code, you have set Reducer to 0, then we do not need the reducer code.
In case you need to clear the output dir before job launch, you can use this snippet to clear the dir if it exists:-
FileSystem fileSystem = FileSystem.get(<hadoop config object>);
if(fileSystem.exists(new Path(<pathTocheck>)))
{
fileSystem.delete(new Path(<pathTocheck>), true);
}
I'm trying to connect from a Google Endpoints server to a Google Cloud SQL server. I'm modifying the Greetings.getGreeting() method in this tutorial:
https://cloud.google.com/appengine/docs/java/endpoints/getstarted/backend/helloendpoints
to call the Cloud mysql database as demonstrated in this tutorial (see doGet method):
https://cloud.google.com/appengine/docs/java/cloud-sql/#enable_connector_j
I have made sure that I can connect to the database from my machine mysql client. The database instance "simple" has a single table "simpletable" who's rows hold an entityID and a string. (But I'm not able to connect, so that's not too important yet.)
This is my endpoints code:
package com.example.helloendpoints;
import com.google.api.server.spi.config.Api;
import com.google.api.server.spi.config.ApiMethod;
import com.google.api.server.spi.response.NotFoundException;
import com.google.appengine.api.users.User;
import java.sql.*;
import java.util.ArrayList;
import javax.inject.Named;
/**
* Defines v1 of a helloworld API, which provides simple "greeting" methods.
*/
#Api(
name = "helloworld",
version = "v1",
scopes = {Constants.EMAIL_SCOPE},
clientIds = {Constants.WEB_CLIENT_ID,
Constants.ANDROID_CLIENT_ID,
Constants.IOS_CLIENT_ID,
Constants.API_EXPLORER_CLIENT_ID},
audiences = {Constants.ANDROID_AUDIENCE}
)
public class Greetings {
public static ArrayList<HelloGreeting> greetings = new ArrayList<HelloGreeting>();
static {
greetings.add(new HelloGreeting("hello world!"));
greetings.add(new HelloGreeting("goodbye world!"));
}
public HelloGreeting getGreeting(#Named("id") Integer id) throws NotFoundException {
// pair to use when running local endpoint server
String urlFromDev = "jdbc:mysql://173.194.XXX.90:3306/simple?user=root";
String classForNameFromDev = "com.mysql.jdbc.Driver";
// pair to use when running cloud endpoint server
String classForNameFromCloud = "com.mysql.jdbc.GoogleDriver";
String urlFromCloud = "jdbc:google:mysql://"
+ Constants.PROJECT_ID + ":"
+ Constants.CLOUD_SQL_INSTANCE_NAME +"/"
+ Constants.DATABASE_NAME + "?user=root";
HelloGreeting helloGreeting = new HelloGreeting();
try {
Class.forName(classForNameFromDev);
// Class.forName(classForNameFromCloud);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
try {
Connection connection = DriverManager.getConnection(urlFromDev);
// Connection connection = DriverManager.getConnection(urlFromCloud);
try {
String statement = "Select simplestring from simpletable where entryID = ?";
PreparedStatement preparedStatement = connection.prepareStatement(statement);
preparedStatement.setInt(1, id);
ResultSet resultSet = preparedStatement.executeQuery();
if (!resultSet.wasNull()) {
helloGreeting.setMessage(resultSet.getString("simplestring"));
} else {
throw new NotFoundException("Greeting not found with an index: " + id);
}
} finally {
connection.close();
}
} catch (SQLException e) {
e.printStackTrace();
}
return helloGreeting;
}
#ApiMethod(name = "greetings.multiply", httpMethod = "post")
public HelloGreeting insertGreeting(#Named("times") Integer times, HelloGreeting greeting) {
HelloGreeting response = new HelloGreeting();
StringBuilder responseBuilder = new StringBuilder();
for (int i = 0; i < times; i++) {
responseBuilder.append(greeting.getMessage());
}
response.setMessage(responseBuilder.toString());
return response;
}
#ApiMethod(name = "greetings.authed", path = "hellogreeting/authed")
public HelloGreeting authedGreeting(User user) {
HelloGreeting response = new HelloGreeting("hello " + user.getEmail());
return response;
}
}
I have tried to enable mysql connector/j in my appengine-web.xml
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<use-google-connector-j>true</use-google-connector-j>
<application>backendapitutorial-1XXX</application>
<version>${app.version}</version>
<threadsafe>true</threadsafe>
<system-properties>
<property name="java.util.logging.config.file" value="WEB- INF/logging.properties"/>
</system-properties>
</appengine-web-app>
Whichever way I build+depl0y it (Dev or cloud), I always get
java.sql.SQLException: No suitable driver found for jdbc:mysql://173.194.XXX.90:3306/simple?user=root
or
java.sql.SQLException: No suitable driver found for jdbc:google:mysql://backendapitutorial-XXXX:simple/simple?user=root
(I replaced the real IP and project name with "X"s for this post).
I've already looked at these:
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/dbname
ClassNotFoundException: com.mysql.jdbc.GoogleDriver
What does 'Class.forName("org.sqlite.JDBC");' do?
I'm building with Maven and working on IntelliJ IDE.
Any help is greatly appreciated. Thanks.
I want to start/resume and stop/suspend instances on google compute engine, but it gives "java.lang.UnsupportedOperationException".Is there any alternative way
to perform these operations?
public class Example {
public static void main(String[] args)
{
String provider = "google-compute-engine";
String identity = "****#developer.gserviceaccount.com";
String credential = "path to private key";
String groupName = "newgroup";
credential = getCredentialFromJsonKeyFile(credential);
Iterable<Module> modules = ImmutableSet.<Module> of(
new SshjSshClientModule(),
new SLF4JLoggingModule(),
new EnterpriseConfigurationModule());
ContextBuilder builder = ContextBuilder.newBuilder(provider)
.credentials(identity, credential)
.modules(modules);
ComputeService compute=builder.buildView(ComputeServiceContext.class).getComputeService();
compute.suspendNode("Instance id");
//compute.suspendNodesMatching(Predicates.<NodeMetadata> and(inGroup(groupName)));
System.out.println("suspended");
compute.getContext().close();
}
private static String getCredentialFromJsonKeyFile(String filename) {
try {
String fileContents = Files.toString(new File(filename), UTF_8);
Supplier<Credentials> credentialSupplier = new GoogleCredentialsFromJson(fileContents);
String credential = credentialSupplier.get().credential;
return credential;
} catch (IOException e) {
System.err.println("Exception reading private key from '%s': " + filename);
e.printStackTrace();
System.exit(1);
return null;
}
}
}
Output:
suspending node(node id)
Exception in thread "main" java.lang.UnsupportedOperationException: suspend is not supported by GCE
at org.jclouds.googlecomputeengine.compute.GoogleComputeEngineServiceAdapter.suspendNode(GoogleComputeEngineServiceAdapter.java:251)
at org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.suspendNode(AdaptingComputeServiceStrategies.java:171)
at org.jclouds.compute.internal.BaseComputeService.suspendNode(BaseComputeService.java:503)
at org.jclouds.examples.compute.basics.Example.main(Example.java:79)
It is not directly supported in the portable jclouds ComputeService, but from the ComputeServiceContext you can get the GoogleComputeEngineApi and the InstanceApi, and use the start/stop methods in there.
FYI, there is an ongoing patch to add support for the start/stop operations in the ComputeService: https://github.com/jclouds/jclouds-labs-google/pull/141
You can stop an instance from the API.
POST https://www.googleapis.com/compute/v1/projects/<project>/zones/<zone>/instances/<instance>/stop
Where :
project in URL is you project id.
zone in URL is the name of zone for the request.
instance in URL is the name of instances to stop.
Here's the docs
hi i am doing an jsp project. and i deploy my project on apache tomcat. i use mysql as databese.
when i deploy project on remote server it is run good. but after some hours it gives me sql error. then i go back my apache server and start projecet again it run and after some hours it gives me same sql error again. i dont know the problem. is that caused from my java connection code or it is about mysql server. can some one tell me why it gives me sql error.?
public class ConnectionManager {
private String className = "com.mysql.jdbc.Driver";
private String userName ="username";
private String password = "password";
private String url = "jdbc:mysql://localhost:3306/database?useUnicode=true&characterEncoding=utf-8";
/**
* #uml.property name="connectionInstance"
* #uml.associationEnd
*/
private static ConnectionManager connectionInstance = null;
public ConnectionManager(){
}
public static synchronized ConnectionManager getInstance() {
if(connectionInstance == null) {
connectionInstance = new ConnectionManager();
}
return connectionInstance;
}
public Connection getConnection(){
Connection conn = null;
try {
Class.forName(className);
conn = DriverManager.getConnection (url, userName, password);
System.out.println("Connection Established");
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
}
return conn;
}
MySQL has a default connection timeout of 8 hours. So this means that you've kept a SQL connection open for too long. Your code suggests that you're creating only one connection on application's startup and reusing it application wide. This is very bad. This is not threadsafe.
You need to change your code so that you're not declaring and storing the SQL Connection as a static or instance variable anywhere in your code. Instead, it should be declared, created and closed within the shortest possible scope. Preferably within the very same method block as where you're executing the SQL query.
Here's a minor rewrite of your ConnectionManager which does the job properly:
public class ConnectionManager {
private static final String DRIVER = "com.mysql.jdbc.Driver";
private static final String USERNAME ="username";
private static final String PASSWORD = "password";
private static final String URL = "jdbc:mysql://localhost:3306/database?useUnicode=true&characterEncoding=utf-8";
static {
try {
Class.forName(DRIVER);
}
catch (ClassNotFoundException e) {
throw new ExceptionInInitializerError(DRIVER + " missing in classpath!", e);
}
}
public static Connection getConnection() throws SQLException {
return DriverManager.getConnection(URL, USERNAME, PASSWORD);
}
}
Use it as follows:
public class SomeDAO {
public SomeEntity find(Long id) throws SQLException {
Connection connection = null;
// ...
try {
connection = ConnectionManager.getConnection();
// ...
}
finally {
// ...
if (connection != null) try { connection.close(); } catch(SQLException ignore) {}
}
return someEntity;
}
To improve connecting performance, use a connection pool instead of DriverManager.
See also:
Show JDBC ResultSet in HTML in JSP page using MVC and DAO pattern
Are you closing connections properly after using them.