Titan 1.0.0 FIXED configuration "set-vertex-id" - can be overriden? - configuration

I'm trying to override the graph.set-vertex-id configuration in Titan 1.0.0
By loading the configuration from an xml file:
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
graph.set-vertex-id=true
storage.batch-loading=true
....
I'm getting the error :
Local setting graph.set-vertex-id=true (Type: FIXED) is overridden by globally managed value (false). Use the ManagementSystem interface instead of the local configuration to control this setting.
I also tried
BaseConfiguration configuration = new BaseConfiguration();
configuration.setProperty("storage.hostname", "any_hostname");
configuration.setProperty("storage.backend", "thrift");
configuration.setProperty("graph.set-vertex-id", Boolean.TRUE);
titanGraph = TitanFactory.open(configuration);
But by checking the config with TitanManagement
titanGraph.openManagement().get("graph.set-vertex-id");
I can see that it's still 'false'.
Has anyone tried to assign custom vertices ids in this version of Titan? Before the upgrade we used Titan 0.4.2 and it worked just fine.
The documentation (http://s3.thinkaurelius.com/docs/titan/1.0.0/configuration.html ) says that
GLOBAL: These options are always read from the cluster configuration and cannot be overwritten on an instance basis.
FIXED: Like GLOBAL, but the value cannot be changed once the Titan cluster is initialized.
When the first Titan instance in a cluster is started, the global configuration options are initialized from the provided local configuration file. Subsequently changing global configuration options is done through Titan’s management API.
To access the management API, call g.getManagementSystem() on an open Titan instance handle g. For example, to change the default caching behavior on a Titan cluster:
mgmt = graph.openManagement()
mgmt.get('cache.db-cache')
// Prints the current config setting
mgmt.set('cache.db-cache', true)
// Changes option
mgmt.get('cache.db-cache')
// Prints 'true'
mgmt.commit()
// Changes take effect
Firstly, there is no getManagementSystem() on a graph in Titan 1.0.0. Secondly, overriding the "graph.set-vertex-id" config like that will throw another error, because at this point the graph is already initialised:
Cannot change the fixed configuration option: root.graph.set-vertex-id
Thank you !

Related

Attribute is required but not set (Unable to create snapshot class for interface)

My Liferay 7 server was using SomeModule happily, until I deployed a new version of SomeModule which has an additional required field favoriteColor.
Now whenever I try to load the portlet Liferay says:
java.lang.RuntimeException: Unable to create snapshot class for interface some.SomeModuleConfiguration
at com.liferay.portal.configuration.metatype.bnd.util.ConfigurableUtil._createConfigurableSnapshot(ConfigurableUtil.java:77)
at com.liferay.portal.configuration.metatype.bnd.util.ConfigurableUtil.createConfigurable(ConfigurableUtil.java:51)
at some.SomeModule.activate(SomeModule.java:50)
...
aused by: java.lang.IllegalStateException: Attribute is required but not set favoriteColor
at aQute.bnd.annotation.metatype.Configurable$ConfigurableHandler.invoke(Configurable.java:75)
at com.sun.proxy.$Proxy1220.favoriteColor(Unknown Source)
at some.SomeModuleConfigurationSnapshot407.<init>(Unknown Source)
The configuration UI for SomeModule does not show anything about favoriteColor.
How to fix that, for instance by setting favoriteColor to its default value?
An alternative path would be using a OSGi configuration file to set defaults and missing values. You can use those files as you do for those modules that come with liferay; e.g., elasticsearch config. (check your osgi/configs directory)
If you are lucky enough to have the source code of the module, you can solve this problem like this:
Temporarily make the new field optional, but replacing required = true to required = false in SomeModuleConfiguration.java.
Deploy the module.
Load the configuration page, save.
Restore to required = true.
Deploy again.
Alternative answers welcome!

Configuration management in builds without source controlled config transformations

Given a configuration named "Data:ConnectionString" in appsettings.json file (ASP.NET Core application), how do I override this in the build? By overriding it can either be that there is a step which changes the value in appsettings.json before compilation during build, or that I override the parameter when using "dotnet test", or something else.
More info:
I have a ASP.NET Core application with standard configuration in appsettings.json. I do not want any connection string or sensitive data checked in the source control.
I am building my application using Visual Studio Team Service (cloud TFS). There is a step where tests are executed, and I want these tests to run against a remote service for which I do not want to check in the credentials.
There are a number of extensions available on http://marketplace.visualstudio.com that will help you without any complicated ness.
https://marketplace.visualstudio.com/items?itemName=YodLabs.VariableTasks
I like the Variable Tasks Pack that comes with:
Set Variable Set a variable value and optionally apply a transformation to it.
Set Variables with Credential Sets the username/password from an existing service endpoint
Set Variable from JSON Extracts a value from JSON using JSONPath
Set Variable from XML Extracts a value from XML using XPath
Update Build Number Allows you to change a build number
Increment Version Increments a semver version number
Super easy... You can also just search for "json" or "variable" to find other options...
Most popular ways:
Use app secrets
Use scripts section in your project.json. You have 4 events -
precompile, postcompile, prepublish, postpublish
You can set the an environmental variable ASPNETCORE_ENVIRONMENT in the build to something like "Test". Create an appsettings.json file named appsettings.Test.Json. Then when you are setting up your configuration in Startup.cs do something like...
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);
When the environmental variable is set to TEST, you new appsettings file will be loaded and can set the connection string to whatever you want.

Warn (or fail) if a package is run without having overriden every pkg connectionstring with a config file entry

It seems like a very common issue with SSIS packages is releasing a package to Production that ends up with running the wrong connectionstring parameters. This could happen by making any one of many mistakes or ommisions. As a result, I find it helpful to dump all ConnectionString values to a log file. This helps me understand what connectionstrings were actually applied to the package at run time.
Now, I am considering having my packages check to see if every connnection object in my package had its connectionstring overriden by an entry in the config file and if not, return a warning or even fail the package. This is to allow easier configuration by extracting all environment variables to a config file. If a connectionstring is never overridden, this risks that a package, when run in production, may use development settings or a package, when run in a non production setting when testing, may accidentily be run against production.
I'd like to borrow from anyone who may have tried to do this. I'd also be interested in suggestions on how to accomplish this with minimal work.
Thx
Technical question 1 - what are my connection string
This is an easy question to answer. In your package, add a Script Task and enumerate through the Connections collection. I fire the OnInformation event and if I had this scheduled, I'd be sure to have the /rep iew options in my dtexec to ensure I record Information, Errors and Warnings.
namespace TurnDownForWhat
{
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
bool fireAgain = false;
foreach (var item in Dts.Connections)
{
Dts.Events.FireInformation(0, "SCR Enumerate Connections", string.Format("{0}->{1}", item.Name, item.ConnectionString), string.Empty, 0, ref fireAgain);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
Running that on my package, I can see I had two Connection managers, CM_FF and CM_OLE along with their connection strings.
Information: 0x0 at SCR Enum, SCR Enumerate Connections: CM_FF->C:\ssisdata\dba_72929.csv
Information: 0x0 at SCR Enum, SCR Enumerate Connections: CM_OLE->Data Source=localhost\dev2012;Initial Catalog=tempdb;Provider=SQLNCLI11;Integrated Security=SSPI;
Add that to ... your OnPreExecute event for all the packages and no one sees it but every reports back.
Technical question 2 - Missed configurations
I'm not aware of anything that will allow a package to know it's under configuration. I'm sure there's an event as you will see in your Information/Warning messages that a package attempted to apply a configuration, didn't find one and is going to retain it's design time value. Information - I'm configuring X via Y. Warning - tried to configure X but didn't find Y. But how to have a package inspect itself to find that out, I have no idea.
That said, I've seen reference to a property that fails package on missed configuration. I'm not seeing it now, but I'm certain it exists in some crevice. You can supply the /w parameter to dtexec which treats warnings as errors and really, warnings are just errors that haven't grown up yet.
Unspoken issue 1 - Permissions
I had a friend who botched an XML config file as part of their production deploy. Their production server started consuming data from a dev server. Bad things happened. It sounds like you have had a similar situation. The resolution is easy, insulate your environments. Are you using the same service account for your production class SQL Server boxes and dev/test/uat/qa/load/etc? STOP. Make a new one. Don't allow prod to talk to any boxes that aren't in their tier of service. Someone bones a package and doesn't set a configuration? First of all, you'll catch it when it goes from dev to something-before-production because that tier wouldn't be able to talk to anything else that's not that level. But if you're in the ultra cheap shop and you've only got dev and prod, so be it. Non-configured package goes to prod. Prod SQL Agent fires off the package. Package uses default connection manager and fails validation because it can't talk to the dev sales database.
Unspoken issue 2 - template
What's your process when you have a new package to build? Does your team really start from scratch? There are so many ways to solve this problem but the core concept is to define your best practices for Configuration, Logging, Package Protection Level, Transaction levels, etc into some easily consumable form. Maybe that's 3 starter packages: one for raw acquisition, maybe one stages and conforms the data and the last one moves data from conformed into the final destination. Teammates then simply have to pick one to start from and fill in the spots that need it. If they choose to do their own thing, that's the stick you beat them with when their package fails to run in production because they didn't follow the standard path.
There are other approaches here. If you're a strong .NET crew, you can gen your template packages that way. At this point, I create my templates with Biml and use that to drive basic package creation.
If I am understanding you correctly the below solution should work.
My suggestion to you is to turn on the Do not save sensitive option for the ProtectionLevel property at the top level of the package.
This will require you to use package configurations for every connection, otherwise it will not have the credentials to make a connection.

setting hadoop job configuration programmatically

I am getting OOM exception (Java heap space) for reduce child. I read in the documentation that increasing the value of mapred.reduce.child.java.opts to -Xmx512M or more would help. Since I am not the admin, I cannot change that value in mapred-site.xml. I would like to set that value only for my job through the java program. I tried setting it using Configuration class as follows, but that didn't work.
Configuration config = new Configuration();
config.set("mapred.reduce.child.java.opts", "-Xmx512M");
JobConf conf1 = new JobConf(config, this.getClass());
The version of Hadoop is 1.0.3
What is the proper way of setting the configuration values programmatically?
AS #ThomasJungblut and #octo have pointed out, the procedure I mentioned in the question is the right way of doing it. The OOM exception still persists, so I would start a new thread instead of continuing here.

OpenGL VBO error causing system exit

I'm using JOGL with to load an OBJ model and display it in a GL canvas using a VBO. Everything is work for the most part however, there are some models where the vertices must be deformed. For example, I have an arrow object and must be able to deform the stem of the arrow to make the tail as long/short as needed while maintaining the object geometry for the arrow head.
This works fine for one instance of the renderer but when I try and add another one to the scene, the system exits on the GLDrawElements call and outputs this error log. Can anyone point me in the right direction? I'm at a complete loss.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000069e3e4c8, pid=6544, tid=2692
#
# JRE version: 6.0_21-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode windows-amd64 )
# Problematic frame:
# C [nvoglnt.dll+0x93e4c8]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
...
Stack: [0x0000000052640000,0x0000000052740000], sp=0x000000005273ecb0, free space=3fb0000000000000000k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [nvoglnt.dll+0x93e4c8]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J com.sun.opengl.impl.GLImpl.glDrawElements0(IIIJ)V
J com.sun.opengl.impl.GLImpl.glDrawElements(IIIJ)V
j com.sonogenics.model.AbstractModelHandler$Renderer.display(Ljavax/media/opengl/GL;)V+196
j com.sonogenics.model.AbstractModelHandler$Renderer.display(Ljavax/media/opengl/GL;Lcom/sonogenics/camera/SimpleProjection;FFFLcom/sonogenics/playout/Field;)V+436
...
Use GDebugger to see what call causes the error and check for invalid data in your gl calls.
It's quite awesome. :)
ACCESS_VIOLATION means you told GL to read memory that is outside the the 'good' areas :)
Within Drawelements there are a couple reasons that could be, you want to check where you setup the GL buffers as well as what you are passing into DrawElements.
-One of your buffers was a bad address, causing it to read from who
knows where
-One of your offsets, strides, were too long causing GL to go beyond the
end of an allocation
-You number of verts you said were in the model was too long... causing it
to go beyond the end of the
allocation
-Your VBO allocation wasn't large enough for the stride * number of
verts