Unable to setup Hadoop in pseudo mode - configuration

I have set up Hadoop on my computer in pseudo-distributed mode.
I followed the directions in Appendix A of 'Hadoop - A Definitive Guide' book to setup Hadoop in a pseudo-distributed mode.
However, from the output of following program, it is safe to infer that my Hadoop is running into standalone mode (i.e. local mode).
public static void main(String[] args) {
Configuration conf = new Configuration();
System.out.println(conf);
System.out.println(conf.get("fs.default.name"));
}
Output:
Configuration: core-default.xml, core-site.xml
file:///
The output is file:/// instead of hdfs://localhost. However the properties in core-site.xml are properly set:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost/</value>
</property>
</configuration>
Also when I submit a test job from Eclipse, it doesn't show up in jobTracker browser UI, I read somewhere that it is due to the fact that Hadoop is running in local mode.
Please let me know what's wrong in my configuration and how I can enable pseudo-distributed mode. Why am I not able to override fs.default.name property in default XML file with that I specified in core-site.xml file?

How are you launching the program? If you're not using the bin/hadoop script then the configuration files in conf/*.xml will not be on the classpath, and hence any values in them will be ignored.
You should also use the ToolRunner launcher:
public class MyJobDriver extends Configured implements Tool {
public static void main(String args[]) {
ToolRunner.run(new MyJobDriver(), args);
}
public int run(String args[]) {
Job job = new Job(getConf());
Configuration conf = job.getConfiguration();
System.out.println(conf);
System.out.println(conf.get("fs.default.name"));
return 0;
}
}
Some other points to note from this code:
Remember to create your Job with the Configuration provided by getConf() - this allows you to use the Generic Options Parser to parse out some common command line switches (-files, -jt, -fs, =Dkey=value etc)
If you need the Configuration to set some custom parameters - get the job copy using job.getConfiguration() - as Job makes a deep copy when you construct it, and any changes to the original will not be applied when you job runs
Then ensure you job is run using the bin/hadoop script:
#> bin/hadoop MyApp.jar a.b.c.MyAppDriver
If you're lauching from Eclipse, ensure the $HADOOP_HOME/conf folder is on the classpath and than will ensure the xml conf files are on the classpath when the Configuration object is created by the ToolRunner.

Related

'DbContextOptionsBuilder' does not contain a definition for 'UseSqlServer'

I am trying to create a Web API in VS 2015 Pro (Update 3) using C# and targeting .NET Core.
I am following this tutorial.
However, I am connecting to a MySQL database instead of an SQL Server database - not sure how much difference that makes...
Anyway, in the tutorial, I have to "Register my context with dependency injection" - so I have to add the following line to the ConfigureServices section of the Startup.cs file:
var connection = Configuration.GetConnectionString("DefaultConnection");
services.AddDbContext< Models.PropWorxContext > (options => options.UseSqlServer(connection));;
However, VS gives me the following error:
Error CS1061 'DbContextOptionsBuilder' does not contain a definition for 'UseSqlServer' and no extension method 'UseSqlServer' accepting a first argument of type 'DbContextOptionsBuilder' could be found (are you missing a using directive or an assembly reference?)
Any ideas why?
This is what the whole Startup.cs file looks like:
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
namespace PropWorxAPI
{
public class Startup
{
public Startup(IHostingEnvironment env)
{
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
.AddEnvironmentVariables();
Configuration = builder.Build();
}
public IConfigurationRoot Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
// Add framework services.
services.AddMvc();
var connection = Configuration.GetConnectionString("DefaultConnection");
services.AddDbContext< Models.PropWorxContext > (options => options.UseSqlServer(connection));
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
loggerFactory.AddConsole(Configuration.GetSection("Logging"));
loggerFactory.AddDebug();
app.UseMvc();
}
}
}
I have also added the MySQL package using the Package Manager, so that my project.json file contains this entry:
*"MySql.Data.EntityFrameworkCore": "7.0.6-IR31"*
Any hints as to where I've gone wrong would be greatly appreciated, as I've spent all day trying to figure it out :( Thank you...
SqlServer is Microsoft Sql Server not MySql, to use SqlServer you would need the package "Microsoft.EntityFrameworkCore.SqlServer": "1.0.*".
If using MySql there is options.UseMySql
see this tutorial for more
I got this error as well whilst setting up a project using Sql Server for a database. In my case I had to add a using statement manually -
"using Microsoft.EntityFrameworkCore;" . It sounds a bit obvious but it threw me as Visual Studio wasn't adding the using statement via Quick Actions.
If you have already installed the "Microsoft.EntityFrameworkCore.SqlServer" package and using Ctrl + Space cannot give you any option, adding the "using Microsoft.EntityFrameworkCore" manually solved my issue.
missing package.This worked for me.
tools-NUget package-package manager-copy paste following install:
Install-Package Microsoft.EntityFrameworkCore.SqlServer -Version 3.1.1
While in your project folder type this into your command line:
dotnet add package Pomelo.EntityFrameworkCore.MySql -v 2.1.2
Install Sqllite/Sqlserver compact toolbox then create database if not created then change connectionstring in appsettings.json

SQLException during Scada-LTS startup [duplicate]

What is wrong with the code there are lots of error while debugging. I am writing a code for a singleton class to connect with the database mysql.
Here is my code
package com.glomindz.mercuri.util;
import java.sql.Connection;
import java.sql.Driver;
import java.sql.DriverManager;
import java.sql.SQLException;
public class MySingleTon {
String url = "jdbc:mysql://localhost:3306/";
String dbName = "test";
String driver = "com.mysql.jdbc.Driver";
String userName = "root";
String password = "";
private static MySingleTon myObj;
private Connection Con ;
private MySingleTon() {
System.out.println("Hello");
Con= createConnection();
}
#SuppressWarnings("rawtypes")
public Connection createConnection() {
Connection connection = null;
try {
// Load the JDBC driver
Class driver_class = Class.forName(driver);
Driver driver = (Driver) driver_class.newInstance();
DriverManager.registerDriver(driver);
connection = DriverManager.getConnection(url + dbName);
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
} catch (InstantiationException e) {
e.printStackTrace();
}
return connection;
}
/**
* Create a static method to get instance.
*/
public static MySingleTon getInstance() {
if (myObj == null) {
myObj = new MySingleTon();
}
return myObj;
}
public static void main(String a[]) {
MySingleTon st = MySingleTon.getInstance();
}
}
I am new to java. Please help.
It seems the mysql connectivity library is not included in the project. Solve the problem following one of the proposed solutions:
MAVEN PROJECTS SOLUTION
Add the mysql-connector dependency to the pom.xml project file:
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.39</version>
</dependency>
Here you are all the versions: https://mvnrepository.com/artifact/mysql/mysql-connector-java
ALL PROJECTS SOLUTION
Add the jar library manually to the project.
Right Click the project -- > build path -- > configure build path
In Libraries Tab press Add External Jar and Select your jar.
You can find zip for mysql-connector here
Explanation:
When building the project, java throws you an exception because a file (the com.mysql.jdbc.Driver class) from the mysql connectivity library is not found. The solution is adding the library to the project, and java will find the com.mysql.jdbc.Driver
If you got the error in your IDE(compile-time error), you need to add your mysql-connector jar file to your libs and add this to your referenced library of project too.
If you get this error when you are running it, then probably its because you have not included mysql-connector JAR file to your webserver's lib folder.
Add mysql-connector-java-5.1.25-bin.jar to your classpath and also to your webserver's lib directory. Tomcat lib path is given as an example Tomcat 6.0\lib
Every one has written an answer but I am still surprised that nobody actually answered it by using the best simple way.
The people answer that include the jar file. But, the error will still occur.
The reason for that is, the jar is not deployed when the project is run. So, what we need to do is, tell the IDE to deploy this jar also.
The people here has answered so many times that put that jar file in the lib folder of WEB-INF. That seems okay, but why do it manually. There is simple way. Check the below steps:
Step 1: If you haven't referenced the jar file into the project then, reference it like this.
Right click on the project and go to the project properties.
Then, go to the java build path, then add external jar file via that.
But this will still not solve the problem because adding the external jar via build path only helps in compiling the classes, and the jar will not be deployed when you run the project. For that follow this step
Right click on the project and go to the project properties.
Then, go to the Deployment Assembly then press Add , then go to the java build path entries and add your libraries whether it is jstl, mysql or any other jar file. add them to deployment.
Below are the two pictures which display it.
For Gradle-based projects you need a dependency on MySQL Java Connector:
dependencies {
compile 'mysql:mysql-connector-java:6.0.+'
}
You will have to include driver jar for MySQL MySQL Connector Jar in your classpath.
If you are using Eclipse:
How to add dependent libraries in Eclipse
If you are using command line include the path to the driver jar using the -cp parameter of java.
java -cp C:\lib\* Main
check for jar(mysql-connector-java-bin) in your classpath download from here
JDBC API mostly consists of interfaces which work independently of any database. A database specific driver is required for each database which implements the JDBC API.
First download the MySQL connector jar from www.mysql.com, then:
Right Click the project -- > build path -- > configure build path
In the libraries tab press Add External Jar and select your jar.
For Maven based projects you need a dependency.
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
</dependency>
The driver connector is not in your build path. Configure the build path and point it to the 'mysql-connector-java-5.1.25-bin.jar' (check the version which you are using). Alternatively you can use maven :D
In the project into the folder Libraries-->right click --> Add Library --> Mysqlconnector 5.1
For IntelliJ Idea, go to your project structure (File, Project Structure), and add the mysql connector .jar file to your global library. Once there, right click on it and chose 'Add to Modules'. Hit Apply / OK and you should be good to go.
This needs to be used as of 2021
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.21</version>
</dependency>
Trivial as it may seem in my case netbeans version maven project 7.2.1 was different. There is a folder in the project called dependencies. Right click and then it brings up a popup window where you can search for packages. In the query area put
mysql-connector
It will bring up the matches (it seems it does this against some repository). Double click then install.
It is because the WEB-INF folder does not exist at the location in the sub directory in the error. You either compile the application to use the WEB-INF folder under public_html OR copy the WEB-INF folder in sub folder as in the error above.
The exception can also occur because of the class path not being defined.
After hours of research and literally going through hundreds of pages, the problem was that the class path of the library was not defined.
Set the class path as follows in your windows machine
set classpath=path\to\your\jdbc\jar\file;.
I Understood your problem add this dependency in your pom.xml your problem will be solved,
https://mvnrepository.com/artifact/mysql/mysql-connector-java/5.1.38
If you are using tomcat then along with project directory you should also copy the database connector jar file to tomcat/lib. this worked for me
I'm developing a simple JavaFX11 application with SQLite Database in Eclipse IDE. To generate report I added Jasper jars. Suddenly it throws "THIS" error.
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
It was running good (BEFORE THIS ADDITION). But suddenly!
I'm not using maven or other managers for this simple application. I'm adding jars manually.
I created "User Library" and added my jars from external folders.
PROBLEM OCCURING AREA:
My "User Library" are marked as system library. I just removed the marking. Now its not a system library. "NOW MY PROJECT WORKING GOOD".
DEBUG MYSELF: Tried other things:
AT RUN CONFIGURATION: Try, removing library and add jars one-by-one and see. - here you have to delete all jars one by one, there is no select all and remove in eclipse right now in run configuration. So the error messages changes form one jars to another.
Hope this helps someone.
I was also facing the same problem
Download mysql-connector-java jar file
paste it in the C:\Program Files\Apache Software Foundation\Tomcat 9.0\lib
Hope this work for you also !!
Finally
I solved by two steps :
1 - add the below to pom.xml
<!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.0.8</version>
</dependency>
2 - Download jar file from this URL:https://mvnrepository.com/artifact/mysql/mysql-connector-java/5.0.8
after that put it in your tomcat/lib folder.
I was having the same issue. I was using intellijj IDE for creating the MySQL connection.
Steps to fix it:
Download: mysql-connector.jar( I used 8.0.29).
Go to "file-->project structure -->Libraries-->Click on plus button and select java and select the jar file you downloaded in step 1".
Check the jar file is showing under "External Libraries directory"
4. Now try to create the connection. It will work.
I used this code for creating MySQL connection:
void createConnection() throws SQLException, ClassNotFoundException {
Connection connection=null;
try {
Class.forName("com.mysql.cj.jdbc.Driver");
connection= DriverManager.getConnection("jdbc:mysql://localhost:3306/candelete"
,"root","");
System.out.println("Connection created");
System.out.println("hashcode is: "+connection.hashCode());
}
finally {
System.out.println("here");
System.out.println("hashcode is: "+connection.hashCode());
connection.close();
System.out.println("hashcode now: "+connection.hashCode());
}
}

How do I get log4J to work - I'm getting "package org.apache.log4j does not exist"

i know this may be a newbie qestion, but I'm having issues with setting up Log4J:
I want to run a log4j demo, and here's my code:
import org.apache.log4j.Logger;
import org.apache.log4j.BasicConfigurator;
public class HelloLOG4j {
private static final Logger logger = Logger.getLogger(Hello.class);
public static void main(String argv[]) {
BasicConfigurator.configure();
logger.debug("Hello world.");
logger.info("What a beatiful day.");
}
}
I set my Classpath:
C:\Users\Adel\Downloads\apache-log4j-1.2.17\log4j-1.2.17.jar
in both System and User var's
But when I run my program I still get
errors found:
File: C:\Users\Adel\Desktop\various_topics\JavaProjects\HelloLOG4j.java [line: 2]
Error: package org.apache.log4j does not exist
I know that I set classpath right - if I run cmd line:
C:\Program Files\Java\jdk1.6.0_20>print %LOG4J_HOME%
C:\Users\Adel\Downloads\apache-log4j-1.2.17\log4j-1.2.17.jar is currently bein
g printed
You need to add log4j home to the classpath as the JVM needs the path to the log4j classes
if on windows, you can use
set classpath=%classpath%;%LOG4J_HOME%
On linux/ ubuntu (much better than windows for development & servers)
export classpath=$classpath:$LOG4J_HOME
then run your app after adding other paths to classpath
like
set classpath=%classpath%;c:\users\adel\....
You do not need to add log4JHOME again - as %classpath%; will add to the current classpath.
LOG4J_HOME is not known to Java. It is just used by log4j in case of auto config/default config.
On a side note try using the new log4j2 !
Can you show how you are trying to compile the code?
And also, try adding the log4j.jar to 'lib' directory and compile with the classpath referencing this jar
Just want to remind that don't capitalized Log4j keyword , unlike Logger:
import org.apache.Log4j.Logger; //typo
import org.apache.log4j.Logger; //correct
/usr/share/java/log4j-1.2-api-2.8.2.jar path can be located by issue dpkg -L liblog4j2-java(debian-based) command, then do:
$ sudo javac -cp .:xxx.jar:/usr/share/java/log4j-1.2-api-2.8.2.jar xxx.java

Hadoop Configuration.addDefaultResource() not working

My following code does not produce expected output:
public static void main(String[] args) throws MalformedURLException {
Configuration.addDefaultResource("/home/some_user/conf.xml");
Configuration conf = new Configuration();
System.out.println(conf);
System.out.println(conf.get("color"));
assertThat(conf.get("color"), is("yellow"));
}
The property color is set in conf.xml file as follows:
<property>
<name>color</name>
<value>yellow</value>
<description>Color</description>
</property>
Looks like file conf.xml isn't getting incorporated in default configuration.
The documentation for Configuration.addDefaultResource(String param) says the param should be in classpath. I don't understand how to add file to the classpath when I am already giving the program full absolute path.
First observation: I don't know which version of hadoop you use but the addDefaultResource() has been deprecated for a very long time.
In the later versions of Hadoop the standard way to accomplish what you want is:
Configuration conf = new Configuration()
conf.addResource("path/to/file");
...
Regarding the classpath issue, you have to simply place the config file in the classpath. So you have to discover what the classpath is to(it is either an environment var or the one which you set with -classpath option). If you didn't use the -classpath option and there is no classpath environment variable then it is automatically set to the current directory (".")

Use external files based on os from junit test when building with hudson/jenkins?

In a maven project I have some junit tests where I need to refer to some runtime libraries (the birt runtime) when running the tests:
#Before
public void setup() {
// init osgi/birt rte.
BEngine.getEngine().init("C:\\birt-runtime-2_6_1\\ReportEngine\\");
}
#Test
public void testname() {
// run test that requires the initialization of the above rte.
}
This works fine when running the tests locally from eclipse. My maven project is also build on a linux server running jenkins but currently I disable tests that requires the above runtime libs.
I am considering to copy the runtime libs to the server running hudson and see if I can get hudson to pick up the location of these files when the tests are executed on hudson.
But to do this I need to use the correct location of the rte in the tests.
Any suggestions for doing this? Eg using env variables that can both be understood on windows/linux?
If the values will not change during the lifetime of the test run (i.e. the build), then use an environmental variable, and just set it as an option in the surefire plugin, systemPropertyVariables.
The correct way to do this for different environments is to use different profiles in maven.
You can have a default profile which includes the set of variables for windows, and another for the hudson.
EDIT: For running the tests correctly from within Eclipse, then simply you can have a default value for the variable
public String getRteLocation() {
String s = System.getProperty("test.rte.location");
return (s == null) ? "C:\\birt-runtime-2_6_1\\ReportEngine\\" : s;
}
#Before
public void setup() {
// init osgi/birt rte.
BEngine.getEngine().init(getRteLocation());
}
or simply set the environment variable in the Run Configuration in Eclipse.
EDIT: Just a clarification, when I say environment variables, I mean the -D variables set from the java command line.