I have a Catboost regression model that I'm saving with onnx to use elsewhere, all using Python. Is it possible to save custom parameters in the onnx model and extract them with onnx runtime? I'd like to save metadata around the model.
Please see https://github.com/onnx/onnx/blob/master/onnx/onnx.proto#L249. You can use this field to store any metadata about the model. This is a free-form data field; hence onnxruntime (ORT) doesn't know anything about it. What do you want ORT to extract and why? Also as a side note, you'll get faster responses if you use https://github.com/Microsoft/onnxruntime/issues. Thanks.
Related
Lately I've been using BIM360Team to store all of my team's .rvt files and then I want to use the room data from the model itself to manipulate some information.
The main goal of using .rvt instead of .nwc for room data extraction is to reduce the re-work as much as possible, So I found this update https://forge.autodesk.com/blog/new-rvt-svf-model-derivative-parameter-generates-additional-content-including-rooms-and-spaces which I really grateful for this update. But it required to translate model again rather than using data management api to get the svf from BIM360Team directly
So is it possible to extract the model data likely using the model derivative api new features? your help would be greatly appreciated.
I am reading a JSON-formatted blob from Azure Storage. I am then using one of the values in that JSON to query a database to get more information. What I need to do is take the JSON from the blob, add the fields from the database to it, then write that combined JSON to another Azure Storage. I cannot, however, figure out how to combine the two pieces of information.
I have tried custom mapping in the copy activity for the pipeline. I have tried parameterized datasets, etc. Nothing seems to provide the results I'm looking for.
Is there a way to accomplish this using native activities and parameters (i.e. not by writing a simple utility and executing it as a custom activity)?
For this I would recommend create a custom U-SQL job to do what you want. So first lookup for both the data you want. Do the job in the U-SQL job and copy the results to the Azure Storage. See this example for your pipeline:
If you are not familiar to U-SQL this can help you:
https://saveenr.gitbooks.io/usql-tutorial/content/
Also this will help you working with Json in your job:
https://www.taygan.co/blog/2018/01/09/azure-data-lake-series-working-with-json-part-2
https://www.taygan.co/blog/2018/03/02/azure-data-lake-series-working-with-json-part-3
I can't find the way that using MySQL database in TensorFlow.
I made the table and sensor data for reference.
Here is the question.
What should I use to read MySQL database in TensorFlow?
I find the way that making CSV file in MySQL, and read it in TensorFlow. But it is not real-time data. I want to use the data in real-time.
Please help me. Thank you.
Late to the party. But i did it like that:
Read data from mysql in chunks (limit+offset) with pandas df.read_sql() in a separate thread.
yield the data with a generator
create a tf.Dataset with .from_tensor_slice() from the pandas dataframe.
Use model.fit() on the dataset.
Get the next chunk of data from the generator and train on that.
So, while the model is training on the data chunk, the next chunk is being loaded in the background. Ideally, the GPU will never sit idle. This can be adjusted by choosing a good chunk size and defining how many epochs the model should be trained on each chunk.
Relevant links:
Make python generator run in background
https://stackoverflow.com/a/29528804/5292996
https://www.tensorflow.org/tutorials/load_data/pandas_dataframe
You'd probably need to just use a standard python interface for mysql and feed it into tensorflow. Here are some examples:
How to read data into Tensorflow?
https://www.quora.com/What-is-the-best-way-to-read-data-into-Tensorflow
I am learning D3.js and want to find some datasets to graph it. Is such kind of dataset is appropriate for the use with D3.js https://data.montgomerycountymd.gov/api/views/54rh-89p8/rows.json?accessType=DOWNLOAD
It seems that data is presented as array without properties -- properties are stored separately in the meta section. Can I convert such data to be used in D3.js? If not, may you please direct me to some resource of simple datasets I can test in D3. Thanks
I work on a large perl website that currently stores all the configurations in a perl module. I have the need to move these settings into MySQL. The problem is the settings are defined in lots of variables and most of them are complex structures (like hash of hashes and array of hashes).
My first idea was to use either XML, YAML, or Storable perl module to easily write and read the variables from a simple file, but my boss doesn't want either of these solutions. He wants it to be stored in MySQL, so other solutions are not an option.
My question is, does anybody know about any CPAN module(s) that will help me to do that task; what I basically need is a way to map all the perl complex perl structures I have to MySQL tables.
Any suggestion would be really appreciated. Thanks!
Option 1: Store the data in serialized form (Data::Dumper, Storable, JSON, etc...) in MySQL's TEXT/MEDIUMTEXT/LONGTEXT type field (65KB/16MB/4GB max sizes respectively)
Option 2: Use DBIx ORM (Object-to-Relational-Mapping), which is the way to automatically map Perl data to DB tables (similar to Java's Hybernate). You'll need to convert your data structures to objects as far as I'm aware, though there may be DBIx module that can deal with non-blessed data structures.
Frankly, unless you need to manipulate the config data in detail within MySQL piece by piece, option #1 is dramatically simpler. However, if your boss's goal is to be able to either query details of configuration, or manipulate its individual elements one by one, you will have to go with #2.
Why you don't want to use Storable qw(freeze thaw) with MySQL?