Integration a deep learning model with ROS - deep-learning

I am using VLP-16 velodyne sensor, I try to detect pedestrians by some deep learning methods.
But I don't know how to use the trained model with ROS.
For example, write a ros node to get the trained model and integrate it with other packages such as velodyne_driver.
I would be grateful if someone could give me some suggestions to achieve it.
Thanks:)

Related

How do you run deep learning models with parsnip?

I have explored the models offered with the R parsnip package listed at https://www.tidymodels.org/find/parsnip/ but I cannot find how to execute a generic deep learning model (by which I mean a deep-layered neural network). The closest I can find are mlp and bag_mlp.
By contrast, I know that the caret package supports at least two deep learning packages: https://topepo.github.io/caret/available-models.html.
Does parsnip not offer deep learning (yet) or am I missing something?

We don't have it via parsnip (at least not yet) since it is difficult to write a static set of tuning parameters for a network of arbitrary size and architecture.
If you want a tidy way to fit basic neural networks with multiple simple layers, the brulee package is helpful. brulee_mlp() can fit those (via torch) and has a recipes interface for easier preprocessing and feature engineering.
Also, though not part of tidymodels, there is the excellent luz package by Daniel Falbel. It's a nice interface to general deep learning models (also via torch).

Is there any deep learning library that gives option to calculate mfcc as the layer of deep learning mode

I want to calculate mfcc as a layer of deep learning model, as Kapre gives option to calculate Log spectrograms or mel spectrograms as a layer of deep learning model. can anybody suggest any deep learning library in Python for this purpose.

You can try torchaudio.transforms.MFCC from the torchaudio package, or librosa.feature.mfcc from the librosa package.

How does the finetune on transformer (t5) work?

I am using pytorch lightning to finetune t5 transformer on a specific task. However, I was not able to understand how the finetuning works. I always see this code :
tokenizer = AutoTokenizer.from_pretrained(hparams.model_name_or_path) model = AutoModelForSeq2SeqLM.from_pretrained(hparams.model_name_or_path)
I don't get how the finetuning is done, are they freezing the whole model and training the head only, (if so how can I change the head) or are they using the pre-trained model as a weight initializing? I have been looking for an answer for couple days already. Any links or help are appreciated.

If you are using PyTorch Lightning, then it won't freeze the head until you specify it do so. Lightning has a callback which you can use to freeze your backbone and training only the head module. See Backbone Finetuning
Also checkout Ligthning-Flash, it allows you to quickly build model for various text tasks and uses Transformers library for backbone. You can use the Trainer to specify which kind of finetuning you want to apply for your training.
Thanks

Using OpenVino pre-trained models with AWS Sagemaker

I'm looking to deploy a pre-trained model for real-time pedestrian and/or vehicle detection using the AWS Sagemaker workflow, I particularly want to use Sagemaker Neo to compile the model and deploy it on the edge. I want to use one of OpenVino's prebuilt models from their model zoo, but when I download the model it is already in their Intermediate Representation (IR) format for their own optimizer.
Is there a way to get an OpenVino pre-trained model not in IR format so that I can use it in sagemaker? Or any possible way to containerize the OpenVino model for use in sagemaker?
If not, are there any free pre-trained models (using any of the popular frameworks like pytorch, tensorflow, ONXX, etc.) that I can use for vehicle detection from a traffic camera POV? AWS Marketplace does not seem to have much to offer in this regard.

Answers to the query as below:
No.Only in Intermediate Representation (IR) format.
There are a few OpenVINO pre-trained models available for vehicle detection.Check out the list of Object Detection Models that are relevant for vehicle detection on these Github pages.
https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/intel/index.md
https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/index.md

what is the type of deep learning algorithm in Rapidminer?

I use rapid-miner's deep learning operator for classification. But I can not find the type of deep learning algorithms.
Does Rapidminer use Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) or others?
Please help me.

RapidMiner provides an H2O integration for multi-layer feed-forward artificial neural networks.
There is also a new Kerras extension available which integrates an interface to the python library.
Regarding the DL4J project, you need to build the extension with gradle and then put the compiled *.jar file into the extension folder in your '.RapidMiner' folder.
Also feel free to ask further, or re-post, questions in the RapidMiner community forum.

This is just a regular MLP (many layers of fully connected neurons), as specified in the documentation, no convolutions, recurrence or anything more complex, just good old MLP.