I'm trying to using Reinforcement learning in CloudSim.
How can I use the Scheduling process in CloudSim as an Environment for the RL model ?
Related
What are the best practices for distributed training with PyTorch custom containers (BYOC) in Amazon Sagemaker? I understand that PyTorch framework supports native distributed training or using the Horovod library for PyTorch.
The recommended approach on Amazon SageMaker is to use the SageMaker built in Data Parallel and Model Parallel Libraries. When you use the Pytorch Deep Learning container provided by SageMaker, the library is built in and you can follow the below examples to get started with examples.
https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-training-notebook-examples.html
If you are bringing your own container, follow the below link to add SageMaker Distributed training support to your container
https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-use-api.html#data-parallel-bring-your-own-container
Apart from this SageMaker also natively supports Pytorch DDP within the native Deep Learning Container used in Pytorch Estimator.
https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html
I have read some paragraph on Self supervision based reinforcement learning which just enables agent to learn without human supervision and effective strategy for unlabeled dataset training. But I found the "Downstream task" many times.
Now, What does it mean by "Downstream task" in Reinforcement Learning?
A downstream task is a task which will be defined via task specific rewards. This is usually done after doing unsupervised RL pretraining for a long time.
Self-Supervision is usually necessary for this Pre-Training,
Most materials (e.g., David Silver's online course) I can find offer discussions about the relationship between supervised learning and reinforcement learning. However, it is actually a comparison between supervised learning and online reinforcement learning where the agent runs in the environment (or simulates interactions) to get feedback given limited knowledge about the underlying dynamics.
I am more curious about offline (batch) reinforcement learning where the dataset (collected learning experiences) is given a priori. What are the differences compared to supervised learning then? and what are the similarities they may share?
I am more curious about the offline (batch) setting for reinforcement learning where the dataset (collected learning experiences) is given a priori. What are the differences compared to supervised learning then ? and what are the similarities they may share ?
In the online setting, the fundamental difference between supervised learning and reinforcement learning is the need for exploration and the trade-off between exploration/exploitation in RL. However also in the offline setting there are several differences which makes RL a more difficult/rich problem than supervised learning. A few differences I can think of on the top of my head:
In reinforcement learning the agent receives what is termed "evaluative feedback" in terms of a scalar reward, which gives the agent some feedback of the quality of the action that was taken but it does not tell the agent if this action is the optimal action or not. Contrast this with supervised learning where the agent receives what is termed "instructive feedback": for each prediction that the learner makes, it receives a feedback (a label) that says what the optimal action/prediction was. The differences between instructive and evaluative feedback is detailed in Rich Sutton's book in the first chapters. Essentially reinforcement learning is optimization with sparse labels, for some actions you may not get any feedback at all, and in other cases the feedback may be delayed, which creates the credit-assignment problem.
In reinforcement learning you have a temporal aspect where the goal is to find an optimal policy that maps states to actions over some horizon (number of time-steps). If the horizon T=1, then it is just a one-off prediction problem like in supervised learning, but if T>1 then it is a sequential optimization problem where you have to find the optimal action not just in a single state but in multiple states and this is further complicated by the fact that the actions taken in one state can influence which actions should be taken in future states (i.e. it is dynamic).
In supervised learning there is a fixed i.i.d distribution from which the data points are drawn (this is the common assumption at least). In RL there is no fixed distribution, rather this distribution depends on the policy that is followed and often this distribution is not i.i.d but rather correlated.
Hence, RL is a much richer problem than supervised learning. In fact, it is possible to convert any supervised learning task into a reinforcement learning task: the loss function of the supervised task can be used as to define a reward function, with smaller losses mapping to larger rewards. Although it is not clear why one would want to do this because it converts the supervised problem into a more difficult reinforcement learning
problem. Reinforcement learning makes fewer assumptions than supervised learning and is therefore in general a harder problem to solve than supervised learning. However, the opposite is not possible, it is in general not possible to convert a reinforcement learning problem into a supervised learning problem.
Due to the cost-saving, I'm running a deep learning model with a regular CPU. It takes 10 seconds to finish a request and it's written in python.
I'm thinking about to improve the perf by using java, C++, or rust. Is there any existing rust framework to pick a deep learning model.
Is there any existing rust framework to pick a deep learning model.
While I am not familiar with rust framework. If you are running you model on intel cpu, I would suggest to export model using ONNX and run it with mxnet with Intel MKLDNN backend. This should give you the most performance as it uses Intel MKLDNN and Intel MKL library. You can use C++/Python.
Install mxnet with MKLDNN
https://mxnet.apache.org/versions/1.6/api/python/docs/tutorials/performance/backend/mkldnn/mkldnn_readme.html
Tensorflow's performance critical parts are written in C++. Using other language won't cause a drastic performance difference. You may Quantize your network or do a Network Pruning to increase performance.
What is the difference between Tensorforce, Kerasrl, and chainerrl used for Reinforcement Learning?
AS far as I've found all three work with OpenAI gym environments and have the same reinforcement learning algorithms that have been implemented. Is there a difference in performance?
they are different Deep learning bank-ends.
TensorFlow , Keras and Chainer are different libraries used for inference of Neural network based AI algorithms.
Open AI is a Reinforcement Learning library.
These are two different technologies.
if you want Reinforcement learning for Tensorflow, back-end and RL tf library, checkout
https://github.com/google/dopamine
This has no connection with OpenAI. Pure Google tech.
short answer: Keras is more "High Level" than tensorflow in the sense that you can write code quicker with Keras but it's less flexible. Checkout this this post for instance.
Tensorflow , keras and Chainer all these are frameworks. These frameworks can be used to implement Deep Reinforcement learning models. As Jaggernaut said Keras is more of high level (meaning : pretty easy to learn) Keras uses Tensorflow backend to function.