When I try to load a big CSV from a zip file, the execution log give me the following error:
----------------------------------------- Error details ------------------------------------------
Component [Clientes:CLIENTES1] finished with status ERROR.
The size of data buffer is only 100663296. Set appropriate parameter in defaultProperties file.
--------------------------------------------------------------------------------------------------
How can I set the appropriate parameter in defaultProperties file?
I tried this link, but my cloudconnect run configurations page is different from the link:
I've created the parameters file and filled the additional parameters with the right values like said the tutorial (code bellow) and the same error appear in the screen.
Name: -config; Value: new_buffer_size.txt
The new_buffer_size.txt content have just this line:DEFAULT_INTERNAL_IO_BUFFER_SIZE = 200000000
How can I solve this problem? I need to solve this before the world explodes.
CloudConnect is designed to develop ETL(s), which can be run on GoodData cloud workers and therefore some lower level settings are surpassed as in this case. The only legitimate way is to modify the ETL the way it can process the data with current settings. Regarding to docs, the referenced article is outdated. GoodData docs team is aware if it and they are preparing docs refactoring.
Note: As you have probably noticed, CloudConnect is being powered by Javlin's Clover ETL, therefore feel free to check their forums, as you would find there how to overcome the issue on lower level (no UI), but it would work only for data processing on the local machine.
I have created a jar file, example.jar, to be consumed by various project at my work place. The jar does have SLF4J for logging, backed by Logback.
However during build , I have excluded logback.xml with assumption that actual application will be having its own logback.xml.
Now , even though actual application does have its logback.xml, logger messages from example.jar are missing .
Could you please guide how to handles multiple logback in above mentioned scenario.
In Play Application , I have added following lines in conf/logback.xml file.
You may disable logging by making level="OFF"
I had read boost doc. But what it described are too limit for my requirement:
My project has a main logger which used to log almost all logs, it use time_based_rotation. Also I want log some message into another file, so I can check those logs independently rather than mix with main logs together.
What I want is:
BOOST_LOG_INLINE_GLOBAL_LOGGER_DEFAULT(gl, boost::log::sources::severity_logger_mt<Logger::severity_level>)
BOOST_LOG(gl::get())<<message; // the main logger
BOOST_LOG_INLINE_GLOBAL_LOGGER_DEFAULT(loggerA, boost::log::sources::logger_mt)
BOOST_LOG(loggerA)<<message; // another logger, log to anther file
The problem is: how should I set the logging core? Add a text_multifile_backend? But the usage is totally different than boost example. I feel in my requirement the file name setting should belong to logger rather than logging core, however I don't know how to do that.
Take a look at this page, You may get some idea.
My program doesn't support thread safe logging while using boost library
Packaging a log4j configuration file in a NetBeans Platform application apparently requires some thinking through. This is what I tried...
I put log4j.xml in src/main/resources/my/package/log4j.xml of some_netbeans_module. The package is a public module package (i.e. classes from this package are used from other packages). I rebuilt the module and confirmed that the file does, in fact, get packaged into the module.
In my classes I get an instance of the logger the way I always do:
static final Logger log = Logger.getLogger(ThisClass.class);
Every NetBeans Platform application has a my_app.conf file which makes it possible to set certain properties. This is where I set log4j.conf:
log4j.configuration="/my/package/log4j.xml"
Now, when I run the application, I see the following output:
[INFO] /home/me/my_app/application/target/my_app/bin/../etc/my_app.conf: 5:
log4j.configuration=/my/package/log4j.xml: not found
What is wrong with the above configuration?
In the my_app.conf file if you append the log4j.configuration property to the default_options property, like so:
default_options="...<other options> -J-Dlog4j.configuration=my/package/log4j.xml"
then this option will get passed to the JVM. Notice that the log4j property has -J-D appended to it. The -J is used by NetBeans to delineate JVM properties and the -D is used by the JVM to delineate a system property.
Also you can/should drop the quotes and the initial / as the quotes are not necessary and NetBeans will complain if you have the initial /
The other way to do this, and the way that I prefer since it doesn't require editing the .conf file, is to put the log4j.xml file into the default package. If you have other requirements that prevents you from doing this then remember that you must put the log4j.configuration property in the app's platform.properties file while your in dev mode and running the app inside of the IDE. Like so:
run.args.extra=-J-Dlog4j.configuration=my/package/log4j.xml
Edit: For questions regarding NetBeans Platform you might have better luck posting to the NetBeans Platform Users forum.
I need to distribute some sort of static configuration through my application. What is the best practice to do that?
I see three options:
Call application:get_env directly whenever a module requires to get configuration value.
Plus: simpler than other options.
Minus: how to test such modules without bringing the whole application thing up?
Minus: how to start certain module with different configuration (if required)?
Pass the configuration (retrieved from application:get_env), to application modules during start-up.
Plus: modules are easier to test, you can start them with different configuration.
Minus: lot of boilerplate code. Changing the configuration format requires fixing several places.
Hold the configuration inside separate configuration process.
Plus: more-or-less type-safe approch. Easier to track where certain parameter is used and change those places.
Minus: need to bring up configuration process before running the modules.
Minus: how to start certain module with different configuration (if required)?
Another approach is to transform your configuration data into an Erlang source module that makes the configuration data available through exports. Then you can change the configuration at any time in a running system by simply loading a new version of the configuration module.
For static configuration in my own projects, I like option (1). I'll show you the steps I take to access a configuration parameter called max_widgets in an application called factory.
First, we'll create a module called factory_env which contains the following:
-define(APPLICATION, factory).
get_env(Key, Default) ->
case application:get_env(?APPLICATION, Key) of
{ok, Value} -> Value;
undefined -> Default
end.
set_env(Key, Value) ->
application:set_env(?APPLICATION, Key, Value).
Next, in a module that needs to read max_widgets we'll define a macro like the following:
-define(MAX_WIDGETS, factory_env:get_env(max_widgets, 1000)).
There are a few nice things about this approach:
Because we used application:set_env/3 and application:get_env/2, we don't actually need to start the factory application in order to have our tests pass.
max_widgets gets a default value, so our code will still work even if the parameter isn't defined.
A second module could use a different default value for max_widgets.
Finally, when we are ready to deploy, we'll put a sys.config file in our priv directory and load it with -config priv/sys.config during startup. This allows us to change configuration parameters on a per-node basis if desired. This cleanly separates configuration from code - e.g. we don't need to make another commit in order to change max_widgets to 500.
You could use a process (a gen_server maybe?) to store your configuration parameters in its state. It should expose a get/set interface. If a value hasn't been explicitly set, it should retrieve a default value.
-export([get/1, set/2]).
...
get(Param) ->
gen_server:call(?MODULE, {get, Param}).
...
handle_call({get, Param}, _From, State) ->
case lookup(Param, State#state.params) of
undefined ->
application:get_env(...);
Value ->
{ok, Value}
end.
...
You could then easily mockup this module in your tests. It will also be easy to update the process with some new configuration at run-time.
You could use pattern matching and tuples to associate different configuration parameters to different modules:
set({ModuleName, ParamName}, Value) ->
...
get({ModuleName, ParamName}) ->
...
Put the process under a supervision tree, so it's started before all the other processes which are going to need the configuration.
Oh, I'm glad nobody suggested parametrized modules so far :)
I'd do option 1 for static configuration. You can always test by setting options via application:set_env/3,4. The reason you want to do this is that your tests of the application will need to run the whole application anyway at some time. And the ability to set test-specific configuration at that point is really neat.
The application controller runs by default, so it is not a problem that you need to go the application-way (you need to do that anyway too!)
Finally, if a process needs specific configuration, say so in the configuration data! You can store any Erlang-term, in particular, you can store a term which makes you able to override configuration parameters for a specific node.
For dynamic configuration, you are probably better off by using a gen_server or using the newest gproc features that lets you store such dynamic configuration.
I've also seen people use a .hrl (erlang header file) where all the configuration is defined and include it at the start of any file that needs configuration.
It makes for very concise configuration lookups, and you get configuration of arbitrary complexity.
I believe you can also reload configuration at runtime by performing hot code reloading of the module. The disadvantage is that if you use configuration in several modules and reload only one of them, only that one module will get its configuration updated.
However, I haven't actually checked if it works like that, and I couldn't find definitive documentation on how .hrl and hot code reloading interact, so make sure to double-check this before you actually use it.