I am using apache-spark 1.2.0 and would like my user's Linux environment variable $MY_KEY to be made available to my Java job when executed using master=local
In Java land this could be passed in using a -D parameter, but I cannot get this recognized when my driver is launched using spark-submit
I have tried adding this to conf/spark-defaults.conf, but spark will not resolve the environment variable $MY_KEY when it executes my Java job (I see this in my logs)
spark.driver.extraJavaOptions -Dkeyfile="${MY_KEY}"
I have tried adding the same as an argument when calling spark-submit, but this doesn't work either.
The same problem with adding it to conf/spark-env.sh
The only way I have got this to work is by editing the bin/spark-submit script directly which defeats the purpose of having it read from the existing environment variable and will get overwritten when I upgrade spark.
So it looks to me like spark-submit ignores your current users' environment variables and only allows a restricted subset of variables to be defined in it's conf files. Does anyone know how I can resolve this?
Related
I would like to change the default Kubernetes log format to "json" for system logs at "create" or preferably at "runtime".
The documentation designates the --logging-format=json for this purpose. However, I have been unable to identify what command-line call I would use to specific this flag.
I have tried kube-apiserver and kind cmd-line with no luck so far.
I am currently using Kind, but any reference related to Kube would be fine.
Questions:
Does this need be at "create" for the cluster?
If runtime is ok, what cmd util would I use to change a cluster configuration at runtime?
Do I need to apply a configuration with "kubectl" with a yaml file?
I am begining to work with ORIENTDB and I have the following question.
Is it mandatory to set both environment variables? I was hoping to work with the studio without them, just setting the XML with my own environment variables.
Is there anyway to use custom variables programatically in my Java program?
Regards.
You don't need to set environment variables to work with OrientDB unless you're planning to run it from outside of its /bin directory, such as a service.
OrientDB Docs | Windows Service
OrientDB Docs | Unix Service
I finally understood what is missing.
When the "OServerPluginManager" is at startup, it uses the ORIENTDB_HOME setted + "plugins" to check the directory and register the plugins.
But between setting the plugin directory using the environment variable and registering the plugins, there is a overriding of properties checking the Server Property "plugin.directory".
So adding the property at server level with the directory where the plugins are will fix the problem.
Is it possible to convert the standalone.xml (or other configured Wildfly/JBoss profile XML file) to a series of commands or script that can be run by jboss-cli.sh? I have a Wildfly 11 instance that I've made config changes to. I'd like to be able to "templatize" it and have the configuration duplicated using shell scripts during my server deployment. Is there a way to export that config as jboss-cli.sh commands?
I haven't tried it on wildfly 11, but previously on wildfly 9 and 10 i've used https://github.com/tfonteyn/profilecloner to generate jboss-cli scripts for profile creation from scratch. The result still required manual intervention, because cli script sometimes broke order of added elements.
Also, due to bugs in Wildfly 10, adding some subsystems from scratch in jboss-cli was not possible - root element refused to be added without subelement, and vise versa (unfortunately i've lost a ticket number where issue was tracked).
Since in my environment we are using domain mode, we started to copy pre-configured profile with /profile=template-name:clone(to-profile=new-profile), but that's irrelevant in standalone case.
I'm going through the "Getting Started with Openshift" guide trying to learn this stuff. I have successfully added and configured a postgres db in my python gear.
If I run an env on the gear itself, I see the database environment variables:
[]\> env |grep POSTGRESQL_DB
OPENSHIFT_POSTGRESQL_DB_HOST=x.x.x.x
OPENSHIFT_POSTGRESQL_DB_USERNAME=admindb
OPENSHIFT_POSTGRESQL_DB_PORT=5432
But in my deployed code, I dump all the env variables with logging.error(os.environ) and none of the POSTGRESQL_DB variables are set. All the python ones seem to load properly.
I know I could set these vars manually, but since these vars are controlled by the gear they could become stale. The documentation implies that these variables should be set (they are used in the sample code).
Found the problem.
In the instructions we set a hot_deploy marker (so the application doesn't bounce when code changes are committed). The next step in the instructions is to add the postgres cartridge, but the environment variables won't load until the cartridge is restarted. So bounce the application and all is well.
Thanks,
Ryan
Is your deployed code running through the same logged-in shell environment as you log in with?
A common solution to making code see shell environment variables that are launched via some non-shell method is to launch them through a login shell.
For example:
/bin/bash -lc "/whatever/you/are/launching --args stuff"
Wondering if it's possible to move the module directory in a JBoss 7 install to a non-default loco.
Does anyone know of a config param to specify where to pick it up?
Kinda like a conf-dir, bin-dir type of thing.
Thanks,
Aaron.
Yes, it's actually possible. As the documentation states, from within the standard launch scripts users are able to manipulate the module path by setting the $JBOSS_MODULEPATH environment variable. (If not set, $JBOSS_MODULEPATH is set to $JBOSS_HOME/modules). The module path is provided to the running process via the -mp command line argument that is set in the standard scripts.