Need one help with MRUnit. I am adding my config file to MapReduceDriver as below.
conf = mapReduceDriver.getConfiguration();
conf.addResource("path_to_config.xml");
When reducer class is trying to access the the property in setUp() mehtod, its not getting the values from the passed in configuration file.
Configuration conf = context.getConfiguration();
String appNameListStr = conf.get("CODE.MAPPING");
// this appNameListStr is returned as null;
Any suggestions/hints on this.
According to the javadocs passing in a String causes the classpath to be examined for a file with that name. You're trying to load a file from the local file system.
You should use addResource(URL url) or addResource(Path file) to look at the local file system.
For example:
conf.addResource(new File("path_to_config.xml").toURI().toURL());
Related
In ASP.NET Core, the JsonConfigurationProvider will load configuration from appsettings.json, and then will read in the environment version, appsettings.{Environment}.json, based on what IHostingEnvironment.EnvironmentName is. The environment version can override the values of the base appsettings.json.
Is there any reasonable way to preview what the resulting overridden configuration looks like?
Obviously, you could write unit tests that explicitly test that elements are overridden to your expectations, but that would be a very laborious workaround with upkeep for every time you change a setting. It's not a good solution if you just wanted to validate that you didn't misplace a bracket or misspell an element name.
Back in ASP.NET's web.config transforms, you could simply right-click on a transform in Visual Studio and choose "Preview Transform". There are also many other ways to preview an XSLT transform outside of Visual Studio. Even for web.config parameterization with Parameters.xml, you could at least execute Web Deploy and review the resulting web.config to make sure it came out right.
There does not seem to be any built-in way to preview appsettings.{Environment}.json's effects on the base file in Visual Studio. I haven't been able to find anything outside of VS to help with this either. JSON overriding doesn't appear to be all that commonplace, even though it is now an integral part of ASP.NET Core.
I've figured out you can achieve a preview with Json.NET's Merge function after loading the appsettings files into JObjects.
Here's a simple console app demonstrating this. Provide it the path to where your appsettings files are and it will emit previews of how they'll look in each environment.
static void Main(string[] args)
{
string targetPath = #"C:\path\to\my\app";
// Parse appsettings.json
var baseConfig = ParseAppSettings($#"{targetPath}\appsettings.json");
// Find all appsettings.{env}.json's
var regex = new Regex(#"appsettings\..+\.json");
var environmentConfigs = Directory.GetFiles(targetPath, "*.json")
.Where(path => regex.IsMatch(path));
foreach (var env in environmentConfigs)
{
// Parse appsettings.{env}.json
var transform = ParseAppSettings(env);
// Clone baseConfig since Merge is a void operation
var result = (JObject)baseConfig.DeepClone();
// Merge the two, making sure to overwrite arrays
result.Merge(transform, new JsonMergeSettings
{
MergeArrayHandling = MergeArrayHandling.Replace
});
// Write the preview to file
string dest = $#"{targetPath}\preview-{Path.GetFileName(env)}";
File.WriteAllText(dest, result.ToString());
}
}
private static JObject ParseAppSettings(string path)
=> JObject.Load(new JsonTextReader(new StreamReader(path)));
While this is no guarantee there won't be some other config source won't override these once deployed, this will at least let you validate that the interactions between these two files will be handled correctly.
There's not really a way to do that, but I think a bit about how this actually works would help you understand why.
With config transforms, there was literal file modification, so it's easy enough to "preview" that, showing the resulting file. The config system in ASP.NET Core is completely different.
It's basically just a dictionary. During startup, each registered configuration provider is run in the order it was registered. The provider reads its configuration source, whether that be a JSON file, system environment variables, command line arguments, etc. and builds key-value pairs, which are then added to the main configuration "dictionary". An "override", such as appsettings.{environment}.json, is really just another JSON provider registered after the appsettings.json provider, which obviously uses a different source (JSON file). Since it's registered after, when an existing key is encountered, its value is overwritten, as is typical for anything being added to a dictionary.
In other words, the "preview" would be completed configuration object (dictionary), which is composed of a number of different sources, not just these JSON files, and things like environment variables or command line arguments will override even the environment-specific JSON (since they're registered after that), so you still wouldn't technically know the the environment-specific JSON applied or not, because the value could be coming from another source that overrode that.
You can use the GetDebugView extension method on the IConfigurationRoot with something like
app.UseEndpoints(endpoints =>
{
if(env.IsDevelopment())
{
endpoints.MapGet("/config", ctx =>
{
var config = (Configuration as IConfigurationRoot).GetDebugView();
return ctx.Response.WriteAsync(config);
});
}
});
However, doing this can impose security risks, as it'll expose all your configuration like connection strings so you should enable this only in development.
You can refer to this article by Andrew Lock to understand how it works: https://andrewlock.net/debugging-configuration-values-in-aspnetcore/
So I'm trying to use this node package. The usage instructions state that I need to require a JSON file. Something like this :
var myschema = require('schema.json');
But when I run this it wants to find a node package by the name of 'schema.json' and throws an error "Cannot find module 'schema.json'". How do I avoid this error?
The name of the libray you linked to is json-schema-deref
As such, to import that library you would do:
var deRefLib = require('json-schema-deref')
Then to load your schema file you'd need to use a require with a relative path. If your schema is in the same directory as your JS file, you would use:
var mySchema = require('./schema.json')
I build a simple weka application using Weka 3.7.13 dev version using MAVEN. I have initialise the pom.xml function to get weka library from the repository.
i want to run a simple clustering algorithm using weka, and connecting it to MySQL database. As per documentation, the weka need to have a DatabaseUtil.props file which contains it's jdbc driver. I have setup that.
This is my project structure:
I can retrieve the DatabaseUtils.props using my code below. But somehow the weka itself cannot recognise the jdbc:Url and driver name which i already configure inside.
DatabaseUtil.props file:
# JDBC driver (comma-separated list)
jdbcDriver=com.mysql.jdbc.Driver
# database URL
jdbcURL=jdbc:mysql://127.0.0.1:3306/datamining
Java code which i trigger the Weka module:
public void dm(){
int cluster = 4;
InstanceQuery clusterQuery = null;
try{
//file reader of database utilities properties
ClassLoader classLoader = getClass().getClassLoader();
File fileProps = new File(classLoader.getResource("DatabaseUtils.props").getFile());
//setup the data preparation functionality then connect to database
clusterQuery = new InstanceQuery();
clusterQuery.setUsername("test");
clusterQuery.setPassword("test");
clusterQuery.setQuery("SELECT * FROM TEMP_KMEANS");
clusterQuery.setCustomPropsFile(fileProps);
But when executed the function returning error: java.sql.SQLException: No suitable driver found for jdbc:idb=experiments.prp
It seems that the DatabaseUtils.props file cannot override the default driver value jdbc:idb=experiments.prp
Any ideas why this is throwing?
Any feedback and answer much appreciated.
I created a DatabaseUtils.props file in the src / main / resources folder and wrote the following code. It worked for me, I'm using weka 3.8.0.
File file = new File(getClass().getClassLoader().getResource("DatabaseUtils.props").toURI());
InstanceQuery query = new InstanceQuery();
query.setCustomPropsFile(file);
query.setUsername(MYSQL_USER);
query.setPassword(MYSQL_PASSWORD);
query.setQuery("select * from Action");
instances = query.retrieveInstances();
System.out.println(instances.toString());
The documentation says "These may be changed by creating a java properties file called DatabaseUtils.props in user.home or the current directory". Your properties file does not match those requirements (a properties file in a jar is not on the filesystem). See also Chapter 14 of the Weka manual.
An alternative solution is to set the URL in code using setDatabaseURL. As long as the JDBC driver is on the classpath this should work.
In my windows store app using the Visual Studio 2012 designer I want to be able to load some model objects for the designer. I've done this plenty of times before where I supply a xaml file using the ms-appx:/// uri without error. However, for this project I need to be able to instantiate a class and have it convert raw xml of a different format into my model objects.
I'm using the following xaml to instantiate my class for the designer:
d:DataContext="{Binding Source={d:DesignInstance Type=model:Walkthroughs, IsDesignTimeCreatable=True}}"
In my Walkthroughs class had code that did this initially:
public Walkthroughs()
{
if (Windows.ApplicationModel.DesignMode.DesignModeEnabled)
AppDataLoader.LoadWalkthroughs(this, XDocument.Load("ms-appx:///SampleData/walkthroughs.xml"));
}
I first ran into an issue where the XDocument.Load did not understand the ms-appx:/// uri so I modified my code to something very simplistic:
AppDataLoader.LoadWalkthroughs(this, XDocument.Load(#"C:\walkthroughs.xml"));
Now I get access to path '' is denied.
I've tried several directories as well to no avail. I'm even running Visual Studio as an Administrator. If I remove the prefix altogether I get the following error:
Could not find file 'C:\Users\{me}\AppData\Local\Microsoft\VisualStudio\11.0\Designer\ShadowCache\omxyijbu.m4y\yofsmg1x.avh\walkthroughs.xml'.
Has anyone been able to load files from the file system when the designer instantiates objects?
Thanks,
-jeff
XDocument.Load(string uri) seems to have problems with loading Project resources from ms-appx:/
Regarding your second approach: Direct access to "C:" is not permitted. Ther is only a handful of special folders that you can access. Check out my workaround for this (my xml file is within the Assets folder of my project:
var storageFolder = Windows.ApplicationModel.Package.Current.InstalledLocation;
storageFolder = await storageFolder.GetFolderAsync("Assets");
var xmlFile = await storageFolder.GetFileAsync("data.xml");
var stream = await xmlFile.OpenReadAsync();
var rdr = new StreamReader(stream.AsStream(), System.Text.Encoding.GetEncoding("ISO-8859-1")); //needed if you have "ä,ß..." in your xml file
var doc = XDocument.Load(rdr);
I am using the EntLib in an environment where database connection strings are retrieved from a separate library call that decrypts a proprietary config file. I have no say over this practice or the format of the config file.
I want to do EntLib exception logging to the database in this setting. I therefore need to set up a EntLib database configuration instance with the name of the database, with the connection string. Since I can't get the connection string until run time, but EntLib does allow run-time configuration, I use the following code, as described in this:
builder.ConfigureData()
.ForDatabaseNamed("Ann")
.ThatIs.ASqlDatabase()
.WithConnectionString(connectionString)
.AsDefault();
The parameter connectionString is the one I've retrieved from the separate library.
The sample code goes on to merge the created configuration info with an empty DictionaryConfigurationSource. I, however, need to merge it with the rest of the configuration code from the app.config. So I do this:
var configSource = new SystemConfigurationSource();
builder.UpdateConfigurationWithReplace(configSource);
EnterpriseLibraryContainer.Current
= EnterpriseLibraryContainer.CreateDefaultContainer(configSource);
... which is based very closely on the sample code.
But: I get an internal error in Microsoft.Practices.EnterpriseLibrary.Common.Configuration.SystemConfigurationSource.Save. The failing code is this:
var fileMap = new ExeConfigurationFileMap { ExeConfigFilename = ConfigurationFilePath };
var config = ConfigurationManager.OpenMappedExeConfiguration(fileMap, ConfigurationUserLevel.None);
config.Sections.Remove(section);
config.Sections.Add(section, configurationSection);
config.Save();
... where 'section' is "connectionStrings". The code fails on the Add method call, saying that you can't add a duplicate section. Inspection shows that the connectionStrings section is still there even after the Remove.
I know from experience that there's always a default entry under connectionStrings when the configuration files are actually read and interpreted, inherited from the machine.config. So perhaps you can never really remove the connectionStrings section.
That would appear to leave me out of luck, though, unless I want to modify the EntLib source, which I do not.
I could perhaps build all the configuration information for the EntLib at run time, using the fluent API. But I'd rather not. The users want their Operations staff to be able to make small changes to the logging without having to involve a developer.
So my question, in several parts: is there a nice simple workaround for this? Does it require a change to the EntLib source? Or have I missed something really simple that would do away with the problem?
I found a workaround, thanks to this post. Rather than taking the system configuration source and attempting to update it from the builder, I copy the sections I set up in app.config into the builder, and then do an UpdateConfigurationWithReplace on an empty dummy configuration source object in order to create a ConfigurationSource that can be used to create the default container.
var builder = new ConfigurationSourceBuilder();
var configSource = new SystemConfigurationSource();
CopyConfigSettings("loggingConfiguration", builder, configSource);
CopyConfigSettings("exceptionHandling", builder, configSource);
// Manually configure the database settings
builder.ConfigureData()
.ForDatabaseNamed("Ann")
.ThatIs.ASqlDatabase()
.WithConnectionString(connectionString)
.AsDefault();
// Update a dummy, empty ConfigSource object with the settings we have built up.
// Remember, this is a config settings object for the EntLib, not for the entire program.
// So it doesn't need all 24 sections or however many you can set in the app.config.
DictionaryConfigurationSource dummySource = new DictionaryConfigurationSource();
builder.UpdateConfigurationWithReplace(dummySource);
// Create the default container using our new ConfigurationSource object.
EnterpriseLibraryContainer.Current
= EnterpriseLibraryContainer.CreateDefaultContainer(dummySource);
The key is this subroutine:
/// <summary>
/// Copies a configuration section from the SystemConfigurationSource to the ConfigurationSourceBuilder.
/// </summary>
/// <param name="sectionName"></param>
/// <param name="builder"></param>
/// <param name="configSource"></param>
private static void CopyConfigSettings(string sectionName, ConfigurationSourceBuilder builder, SystemConfigurationSource configSource)
{
ConfigurationSection section = configSource.GetSection(sectionName);
builder.AddSection(sectionName, section);
}