How do I train AI model through the data available in the internet in python ?
I need to know how arrange the data and train the model
Related
I am working on entity and relation extraction for my knowledge graph and want to use deep learning model for entity and relation extraction from news articles, but I see there is no existing datasets related to maritime news that's why I want to create small dataset like nyt or webnlg to train my model with.
Is there any particular way to create dataset? or which kind of tools or libraries will be needed to process the text and make news dataset?
Trying to build new dataset for maritime news which will be train or used with deep learning extraction model.
I have a Catboost regression model that I'm saving with onnx to use elsewhere, all using Python. Is it possible to save custom parameters in the onnx model and extract them with onnx runtime? I'd like to save metadata around the model.
Please see https://github.com/onnx/onnx/blob/master/onnx/onnx.proto#L249. You can use this field to store any metadata about the model. This is a free-form data field; hence onnxruntime (ORT) doesn't know anything about it. What do you want ORT to extract and why? Also as a side note, you'll get faster responses if you use https://github.com/Microsoft/onnxruntime/issues. Thanks.
I need to replicate and continuously sync a MySQL database to a graph database, to further query and analyze data from an eCommerce application.
Came across DZone Article which talks about syncing data between oracle and Neo4J. They are using Oracle GoldenGate and Apache Kafka for this.
However, I have a few questions on how this would be implemented -
What about the graph data model? Can I generate graph data model from existing RDBMS data model? Are there any open source ETL solutions to map data models between source and target?
The RDBMS data model may change as part of new features. How to keep data models in sync between a source RDBMS and target graph database?
Are there any other approaches to make this work?
I'm fairly new to SQL to NoSQL replication. Any inputs?
I can't find the way that using MySQL database in TensorFlow.
I made the table and sensor data for reference.
Here is the question.
What should I use to read MySQL database in TensorFlow?
I find the way that making CSV file in MySQL, and read it in TensorFlow. But it is not real-time data. I want to use the data in real-time.
Please help me. Thank you.
Late to the party. But i did it like that:
Read data from mysql in chunks (limit+offset) with pandas df.read_sql() in a separate thread.
yield the data with a generator
create a tf.Dataset with .from_tensor_slice() from the pandas dataframe.
Use model.fit() on the dataset.
Get the next chunk of data from the generator and train on that.
So, while the model is training on the data chunk, the next chunk is being loaded in the background. Ideally, the GPU will never sit idle. This can be adjusted by choosing a good chunk size and defining how many epochs the model should be trained on each chunk.
Relevant links:
Make python generator run in background
https://stackoverflow.com/a/29528804/5292996
https://www.tensorflow.org/tutorials/load_data/pandas_dataframe
You'd probably need to just use a standard python interface for mysql and feed it into tensorflow. Here are some examples:
How to read data into Tensorflow?
https://www.quora.com/What-is-the-best-way-to-read-data-into-Tensorflow
I am currently trying to store my deep learning models from tensorflow and keras in a graph database called ArangoDB. Like most object databases, ArangoDB requires the files to be in JSON. I may be willing to switch to HDFS, but either way Tensorflow and Keras insist on using proprietary HDF5 format from the HDF5 group to store their weights.
How can I convert these using python to a JSON format to be stored in the DB and how can I convert them back to be loaded again in tensorflow?
https://machinelearningmastery.com/save-load-keras-deep-learning-models/ shows how you could save models in JSON with keras. Once you have the file in JSON format, you can persist that.
The following does not store a model but stores TFX artifacts in ArangoDB.
https://github.com/arangoml/arangopipe/blob/master/arangopipe/tests/TFX/tfx_metadata_integration.ipynb