Automatically generate R source code to build a package - json

I wrote bindings to an API and put everything into an R package, including tests, vignettes, etc., but the API keeps constantly changing. This brings up some issues
updating my package is error-prone, maybe I miss a new function or forget to mark an old as deprecated
submitting the package to CRAN is not a good idea, since it's changing frequently and packages are reviewed by hand
I got a hard time keeping this software up2date, since the API chance irregularly and therefor I maybe miss them
I came up with the idea to generate the bindings automatically. The API itself provides everything required for that via an online JSON documentation. These docs reflect constantly the current definition of the API.
Writing some code which converts the JSON docs to R functions is not the problem. But if I do so, I still need to update the package on CRAN. The best solution would be, to create a package that (on load) looks up the API definition and creates the required functions. Ideally these functions should be unit tested.
I am thankful for any hint on that.
Best
Edit: The API is the firebrowse API with an example of what the input would be.

This is really challenging and thus there's no obvious way to do it. The whole idea behind wsdl was to be able to do this easily using a standardized XML description. That was never really implemented in R and it never really took off more broadly (because of the emergence of RESTful services and JSON).
You can definitely generate functions dynamically by creating a so-called "function factories" (Hadley discussed these a bit here). In short, you write a function that takes JSON as input and returns a function that does whatever is described in the JSON. (Creating such a factory that dynamically does this whenever the package is loaded seems risky but I suppose it's possible. I'd probably just keep the factory to myself and use it to create and update the package.)
I'm not going to attempt to deal with your API specifically, but to see how this would work:
# create factory with arguments to control returned function
factory <- function(action, endpoint, content = TRUE, parsed = FALSE) {
if (content) {
if(parsed) {
out <- function() httr::content(httr::VERB(action, endpoint))
} else {
out <- function() httr::content(httr::VERB(action, endpoint), "text")
}
} else {
out <- function() httr::VERB(action, endpoint)
}
return(out)
}
# use factory to create different functions
(a <- factory("GET", "http://example.com", content = TRUE, parsed = FALSE))
## function() httr::content(httr::VERB(action, endpoint), "text")
(b <- factory("GET", "http://example.com", content = TRUE, parsed = TRUE))
## function() httr::content(httr::VERB(action, endpoint))
(c <- factory("GET", "http://example.com", content = FALSE))
function() httr::VERB(action, endpoint)
# evaluate each function
a() # returns a character string
b() # returns parsed HTML
c() # returns an httr response object

The best solution would be, to create a package that (on load) looks up the API definition and creates the required functions. Ideally these functions should be unit tested.
This is a very well known problem. React to server changes without breaking the clients is a pain not just in your situation, but also for mobile applications (that needs to be resubmitted every time API changes).
While your approach may work (generate the client on the fly), the best result can be reached if the server may collaborate to reach the achievement.
You have to decouple the client from API implementation. How? Using REST (for real), thous introducing the concept of state and transitions.
This is not the right place to explain how it works, but a great introduction can be found in this great presentation by Glenn Block, and then continuing to read.
This won't solve your particular problem, but it is, in my opinion, the right way to approach the problem.
You may want to have a look to this video as well, 15:24 part.

Related

SolidJS: "computations created outside a `createRoot` or `render` will never be disposed" messages in the console log

When working on a SolidJS project you might start seeing the following warning message in your JS console:
computations created outside a `createRoot` or `render` will never be disposed
There are some information available on this in SolidJS' Github repository issues. But after reading them I was still not quite sure what this was all about and whether my code was really doing something wrong.
I managed to track down where it came from and find a fix for it based on the documentation. So I'm providing the explanation and the solution for those Googling this warning message.
In essence this is a warning about a possibility of a memory leak due to a reactive computation being created without the proper context which would dispose of it when no longer needed.
A proper context is created a couple of different ways. Here are the ones I know about:
By using the render function.
By using the createRoot function. Under the hood render uses this.
By using the createContext function.
The first is by far the most common way, because each app has at least one render function call to get the whole show started.
So what makes the code go "out of context"?
Probably the most common way is via async calls. The context creation with its dependency tree happens only when the synchronous portion of the code finishes running. This includes all the export default function in your modules and the main app function.
But code that runs at a later time because of a setTimeout or by being in an async function will be outside of this context and any reactive computations created will not be tracked and might stick around without being garbage collected.
An example
Let's say you have a data input screen and have a Save button on it that makes an API call to your server to save the data. And you want to provide a feedback to the user whether the operation succeeded or not, with a nice HTML formatted message.
[msg,setMsg] = createSignal(<></>)
async function saveForm(){
...
setMsg(<p>Saving your data.<i>Please stand by...</i></p>)
const result=await callApi('updateUser',formData)
if(result.ok){
setMsg(<p>Your changes were <b>successfully</b> saved!</p> )
} else {
setMsg(<p>There was a problem saving your data! <br>Error: </p><pre>{result.error}</pre> )
}
}
...
<div>
...
<button onClick={saveForm} >Save</button>
{msg()}
</div>
This will produce the above mentioned warning when the API call returns an error, but not the other times. Why?
The reason for this is that SolidJS considers the code inserts inside JSX to be reactive, ie: need to be watched and re-evaluated. So inserting the error message from the API call creates a reactive computation.
The solution
I found the solution at the very end of the SolidJS doc. It's a special JSX modifier: /*#once*/
It can be used at the beginning of a curly brace expression and it tells the SolidJS compiler to explicitly not to make this a reactive expression. In other words: it will evaluated once and only once when the DOM nodes are created from the JSX.
In the above example here's how to use it:
setMsg(<p>There was a problem saving your data! <br>Error: </p><pre>{/*#once*/ result.error}</pre> )
After this there will be no more warning messages :)
In my case, I had an input and when that input changed I re-created an SVG drawing. Because the SVG creation was an expensive operation, I added a debounce in the createEffect function which ran when the input changed. debounce is a technique to defer the processing until the input stops changing for at least X amount of time. It involved running the SVG generation code inside the setTimeout function, thus being outside of the main context. Using the /*#once*/ modifier everywhere where I inserted an expression in the generated JSX has fixed the problem.

Accessing regmap RegFields

I am trying to find a clean way to access the regmap that is used with *RegisterNode for creating documentation and testing files. The TLRegisterNode has methods for generating the json through some Annotations. These are done in the regmap method by adding them to the ElaborationArtefacts object. Other protocols don't seem to have these annotations.
Is there anyway to iterate over the "regmap" Register Fields post elaboration or during?
I cannot just access the regmap as it's not really a val/var since it's a method. I can't quite figure out where this information is being stored. I don't really believe it's actually "storing" any information as much as it is simply creating the hardware to attach the specified logic to the RegisterNode based logic.
The JSON output is actually fine for me as I could just write a post processing script to convert JSON to my required formats, but I'm wondering if I can access this information OR if I could add a custom function call at the end. I cannot extend the case class *RegisterNode, but I'm not sure if it's possible to add custom functions to run at the end of the regmap method.
Here is something I threw together quickly:
//in *RegisterRouter.scala
def customregmap(customFunc: (RegField.Map*) => Unit, mapping: RegField.Map*) = {
regmap(mapping:_*)
customFunc(mapping:_*)
}
def regmap(mapping: RegField.Map*) = {
//normal stuff
}
A user could then create a custom function to run and pass it to the regmap or to the RegisterRouter
def myFunc(mapping: RegField.Map*): Unit = {
println("I'm doing my custom function for regmap!")
}
// ...
node.customregmap(myFunc,
0x0 -> coreControlRegFields,
0x4 -> fdControlRegFields,
0x8 -> fdControl2RegFields,
)
This is just a quick example I have. I believe what would be better, if something like this was possible, would be to have a Seq of functions that could be added to the RegisterNode that are ran at the end of the regmap method, similar to how TLRegisterNode currently works. So a user could add an arbitrary number and you still use the regmap call.
Background (not directly part of question):
I have a unified register script that I have built over the years in which I describe the registers for a particular IP. It works very similar to the RegField/node.regmap, except it obviously doesn't know about diplomacy and the like. It will generate the Verilog, but also a variety of files for DV (basic `defines for simple verilog simulations and more complex uvm_reg_block defines also with the ability to describe multiple of the IPs for a subsystem all the way up to an SoC level). It will also print out C Header files for SW and Sphinx reStructuredText for documentation.
Diplomacy actually solves one of the main issues I've been dealing with so I'm obviously trying to push most of my newer designs to Chisel/Diplo.
I ended up solving this by creating my own RegisterNode which is the same as the rocketchip RegisterNodes except that I use a different Elaboration Artifact to grab the info and store it for later.

Can you preview ASP.NET Core's appsettings.json environment overrides?

In ASP.NET Core, the JsonConfigurationProvider will load configuration from appsettings.json, and then will read in the environment version, appsettings.{Environment}.json, based on what IHostingEnvironment.EnvironmentName is. The environment version can override the values of the base appsettings.json.
Is there any reasonable way to preview what the resulting overridden configuration looks like?
Obviously, you could write unit tests that explicitly test that elements are overridden to your expectations, but that would be a very laborious workaround with upkeep for every time you change a setting. It's not a good solution if you just wanted to validate that you didn't misplace a bracket or misspell an element name.
Back in ASP.NET's web.config transforms, you could simply right-click on a transform in Visual Studio and choose "Preview Transform". There are also many other ways to preview an XSLT transform outside of Visual Studio. Even for web.config parameterization with Parameters.xml, you could at least execute Web Deploy and review the resulting web.config to make sure it came out right.
There does not seem to be any built-in way to preview appsettings.{Environment}.json's effects on the base file in Visual Studio. I haven't been able to find anything outside of VS to help with this either. JSON overriding doesn't appear to be all that commonplace, even though it is now an integral part of ASP.NET Core.
I've figured out you can achieve a preview with Json.NET's Merge function after loading the appsettings files into JObjects.
Here's a simple console app demonstrating this. Provide it the path to where your appsettings files are and it will emit previews of how they'll look in each environment.
static void Main(string[] args)
{
string targetPath = #"C:\path\to\my\app";
// Parse appsettings.json
var baseConfig = ParseAppSettings($#"{targetPath}\appsettings.json");
// Find all appsettings.{env}.json's
var regex = new Regex(#"appsettings\..+\.json");
var environmentConfigs = Directory.GetFiles(targetPath, "*.json")
.Where(path => regex.IsMatch(path));
foreach (var env in environmentConfigs)
{
// Parse appsettings.{env}.json
var transform = ParseAppSettings(env);
// Clone baseConfig since Merge is a void operation
var result = (JObject)baseConfig.DeepClone();
// Merge the two, making sure to overwrite arrays
result.Merge(transform, new JsonMergeSettings
{
MergeArrayHandling = MergeArrayHandling.Replace
});
// Write the preview to file
string dest = $#"{targetPath}\preview-{Path.GetFileName(env)}";
File.WriteAllText(dest, result.ToString());
}
}
private static JObject ParseAppSettings(string path)
=> JObject.Load(new JsonTextReader(new StreamReader(path)));
While this is no guarantee there won't be some other config source won't override these once deployed, this will at least let you validate that the interactions between these two files will be handled correctly.
There's not really a way to do that, but I think a bit about how this actually works would help you understand why.
With config transforms, there was literal file modification, so it's easy enough to "preview" that, showing the resulting file. The config system in ASP.NET Core is completely different.
It's basically just a dictionary. During startup, each registered configuration provider is run in the order it was registered. The provider reads its configuration source, whether that be a JSON file, system environment variables, command line arguments, etc. and builds key-value pairs, which are then added to the main configuration "dictionary". An "override", such as appsettings.{environment}.json, is really just another JSON provider registered after the appsettings.json provider, which obviously uses a different source (JSON file). Since it's registered after, when an existing key is encountered, its value is overwritten, as is typical for anything being added to a dictionary.
In other words, the "preview" would be completed configuration object (dictionary), which is composed of a number of different sources, not just these JSON files, and things like environment variables or command line arguments will override even the environment-specific JSON (since they're registered after that), so you still wouldn't technically know the the environment-specific JSON applied or not, because the value could be coming from another source that overrode that.
You can use the GetDebugView extension method on the IConfigurationRoot with something like
app.UseEndpoints(endpoints =>
{
if(env.IsDevelopment())
{
endpoints.MapGet("/config", ctx =>
{
var config = (Configuration as IConfigurationRoot).GetDebugView();
return ctx.Response.WriteAsync(config);
});
}
});
However, doing this can impose security risks, as it'll expose all your configuration like connection strings so you should enable this only in development.
You can refer to this article by Andrew Lock to understand how it works: https://andrewlock.net/debugging-configuration-values-in-aspnetcore/

How do I retrieve the json representation of an azure data factory pipeline?

I want to track pipeline changes in source control, and I'm looking for a way to programmatically retrieve the json representation from the ADF.
The .Net routines return the objects, but sadly ToString() does not return json (wouldn't THAT be convenient?), so right now I'm looking at copying the json down by hand (shoot me now!), or possibly trying to recreate the json from the .Net objects (shoot me later!).
Please tell me I'm being dense and there is an obvious way to do this.
You can serialize the object using Newtonsoft Json.
See (https://azure.microsoft.com/en-us/documentation/articles/data-factory-create-data-factories-programmatically/) for how to connect via the ADF SDK
var aadTokenCredentials = new TokenCloudCredentials(ConfigurationManager.AppSettings["SubscriptionId"], GetAuthorizationHeader());
var resourceManagerUri = new Uri(ConfigurationManager.AppSettings["ResourceManagerEndpoint"]);
var manager = new DataFactoryManagementClient(aadTokenCredentials, resourceManagerUri);
var pipeline = manager.Pipelines.Get(resourceGroupName, dataFactoryName, pipelineName);
var pipelineAsJson = JsonConvert.SerializeObject(pipeline.Pipeline, Formatting.Indented);
I was expecting something more complex but looking at the sdk source GitHub it is not doing anything special.
Our team has a deployment tool that takes git changes and deploy them appropriately. Everything is done asynchronously and being controlled and versioned through git.
In a nutshell our deployment has the following flow:
Any completed git merge request triggers a VSO build. This is simply
building the whole solution via MsBuild.
Every successful build is applied a Git tag for tracking of Last Known Good.
Next (if build succeeded) our .net ADFPublisher starts by taking only the changed data factory files and asynchronously publishing them based on their
git operation (modified, add, delete, etc.).
For some failures cases our ADFPublisher will perform a retry.
This whole process (Build + publish) takes ~ 65 seconds and has
already saved us from having several bugs. It also allows us to move
definitions from one environment to another very easily.
Let me know if you think this is something that you will be interested in and I will setup a way to share it with you

Why is native .bind() so slow?

I recently came across this issue. For a project I'm working on, we were using .bind() way too often and it actually hit the performance quite hard considering that we only have 16ms for the rendering loop to do things.
So I did some jsperf and noticed that calling a bound function (besides of the extra garbage) is way slower than calling an unbound function or using .call on a function.
I literally changed every piece of code to avoid bindings and to use .call/.apply instead. Ding this i not only spawned less functions but also increased the performance of my app a great deal.
However, I was unsatisfied with this and wrote a new way of binding functions.
https://github.com/SebastianNette/FastBind
This is overwriting the native bind method with a .call/.apply approach.
And it runs 96% faster.
Doing some testings on nodejs is came to these results:
Calling a bound function is 20 times slower than calling an unbound function.
Calling a bound function with my own approach takes only 2 times the time of the unbound call.
So I was wondering what is wrong with the native binding function. Why does it behave like that? And which would be the best way to deal with that issue.
Most of my app code is now written like that:
var scope = this;
this.boundFn = function(a,b,c) { return scope.fn(a,b,c); };
Or even
this.callback = fn;
this.context = context;
this.callback.call(this.context);
I do prefer the latter because it doesn't spawn any new functions. However, sometimes I just do have to bind. (handlers, timers, etc).
My educated guess is that it makes a clone of the object you are using but replaces the underlying prototype of object. Instead of using a generic precompiled object from the page rendered code it now has to take two things:
The passed variable thats to be come this. analyse it, clone it. then inject the specified function thats to be called into the new object. Then execute the function in the new object. afterwards if no longer called clean it up.
The more complex and more scoping loops an object has the long the bind will take because the engine needs to traverse the scope tree of all functions and parameters to see what needs to be copied.
You are already using scoping, which I strongly advice. It is less memory intense and the engine does not have to copy the objects and then call the functions. And you get the added benefit that you can access properties from both objects.
In my experience binding is never truly needed. Just use setters and getters for properties, otherwise the scoped variables won't always change in the main object.
Take for example this snippet
function domagic() {
this.myproperty = "Hello ";
}
domagic.prototype = {
perform:function(){
var that = this;
var hello = "World";
setTimeout(function(){
// this in this contect is whatever runs timeout. not domagic
// I use this for jQuery and my own objects to get best
// of both worlds, but I always post a comment in a scope
// to remind myself what this and that refers to.
window.alert(that.myproperty+hello);
that.set("Goodbye ");
},2000);
},
set : function(what) {
this.myproperty = what;
}
};
magic = new domagic();
magic.perform();
setTimeout(function(){magic.perform();},2000);