I am using Sutton's procedure to represent the continuous state space into new discrete space of another dimensions. Now, I am having hard time figuring out how to create the Q-table.
Let's say that this procedure returns a list of size 64 representing new state from original state of 15 dimensions, what are the dimensions of Q-table if you have 22 actions?
Related
I'm trying to develop a model to recognize new gestures with the Myo Armband. (It's an armband that possesses 8 electrical sensors and can recognize 5 hand gestures). I'd like to record the sensors' raw data for a new gesture and feed it to a model so it can recognize it.
I'm new to machine/deep learning and I'm using CNTK. I'm wondering what would be the best way to do it.
I'm struggling to understand how to create the trainer. The input data looks like something like that I'm thinking about using 20 sets of these 8 values (they're between -127 and 127). So one label is the output of 20 sets of values.
I don't really know how to do that, I've seen tutorials where images are linked with their label but it's not the same idea. And even after the training is done, how can I avoid the model to recognize this one gesture whatever I do since it's the only one it's been trained for.
An easy way to get you started would be to create 161 columns (8 columns for each of the 20 time steps + the designated label). You would rearrange the columns like
emg1_t01, emg2_t01, emg3_t01, ..., emg8_t20, gesture_id
This will give you the right 2D format to use different algorithms in sklearn as well as a feed forward neural network in CNTK. You would use the first 160 columns to predict the 161th one.
Once you have that working you can model your data to better represent the natural time series order it contains. You would move away from a 2D shape and instead create a 3D array to represent your data.
The first axis shows the number of samples
The second axis shows the number of time steps (20)
The thirst axis shows the number of sensors (8)
With this shape you're all set to use a 1D convolutional model (CNN) in CNTK that traverses the time axis to learn local patterns from one step to the next.
You might also want to look into RNNs which are often used to work with time series data. However, RNNs are sometimes hard to train and a recent paper suggests that CNNs should be the natural starting point to work with sequence data.
I'm trying to read variable length 1-D inputs into a Tensorflow CNN.
I have previously implemented reading fixed length inputs by first constructing a CSV file (where the first column is the label and the remaining columns are the input values - flattened spectrogram data all padded/truncated to the same length) using tf.TextLineReader().
This time I have a directory full of files each one containing a line of data I want to use as input (flattened spectrogram data again but I do not want to force them to the same dimensions), and the line lengths are not fixed. I'm getting an error trying to use the previous approach of compiling a CSV first. I looked into the documentation of tf.TextLineReader() and it specifies that all CSV rows must be the same shape, so I am stuck! Any help would be much appreciated, thanks :)
I'm assuming that the data isn't changing shape when you have a longer or shorter sample right? By that I mean that if you trained your network on arrays of 1000 pixels for example, with a kernel of say [5,1] size. That [5,1] kernel needs to see the same patterns in the variable length data as it did in the training data. If your data is stretched or shrunk, then the correct solution is to interpolate the data to the same size as the training data so the shapes/patterns match.
Assuming you just want variable length inputs, then in theory you should be able to do this by setting your batch size to 1 and varying the 1st dimension of the data.
So your input placeholder would look like:
X = tf.placeholder(dtype, shape=[1,None,1,1])
The 4 shape arguments are: 1=batch size; None=unknown first dimension size; 1=unused because it's a 1D dataset, 1=one channel images, again unused but necessary for tf.conv2d to receive the expected 4D image.
This is not very different from configuring tensorflow to support variable batch sizes. So you should review this link below and understand that process.
get the size of a variable batch dimension
Note that you can't use a batch size more than 1 here because you wouldn't be able to construct a matrix with missing values in the 2nd dimension. I expect the convolution operations to work with this variable dimension (though I haven't actually tried this).
Another option to deal with this problem would be to pad your inputs with 0's so they all have a common length, but that will need to have been trained into the model up front.
I have a table for segments storages on table as polygons.
Then I want to get all segments that are touched by another polygon for example a square, or a circle.
On image : http://img.acianetmedia.com/GJ3
I represent small gray boxes as segments and big_BOX .
With this query:
SELECT id, position, ASTEXT( value )
FROM segment
WHERE MBRCONTAINS( GEOMFROMTEXT( 'POLYGON(( 20.617202597319 -103.40838420263,20.617202597319 -103.3795955521,20.590250599403 -103.3795955521,20.590250599403 -103.40838420263,20.617202597319 -103.40838420263))' ) , value )
I got 4 segments that are 100% inside big_BOX,
but how to get ALL segments that are touched by big_BOX ?
result has to be 16 segments.
A simple solution:
Instead of MBRContains, you should use MBRIntersects which will return any results that either fully or partially cross space with your big box.
A caution and full solution:
Dependent on your data, and the rest of your solution (especially on how big box is formed), it may be possible that you return more than 16 segments due the number of decimal places your coordinates use. Whilst this is however quite unlikely and would only ever be possible under extreme circumstances its just a possibility to consider.
At 7 decimal places, you're at 1.1cm accuracy (at the equator). If your big box looked to exactly line up with a 4x4 segment set, it is possible (at an absolute maximum degree) that you actually get a result set of 36 (6x6) due to the coordinates overlapping into the next segment on all sides by even the most minute measurement. Any multiple of 4 between 16 and 36 inclusive could be possible.
Again, this is largely unlikely, but if you wanted to always ensure a result set of 16 you could use a combination of methods such as Area(Intersection(#geom1, #geom2)) to calculate the intersection geography between your big box and Intersecting segments, order on that column descending and take the first 16 results.
Whilst this would guarantee the most appropriate 16 segments, it will add additional overhead to all queries just to cater for the most extreme scenarios.
The choice is yours. Hope it helps.
Given a dataset of samples in a multi dimensional space (in my case a 4D space) where the samples are present on all the corners of the 4D cube and a substantial amount of samples within this cube but not in a neatly grid. Each sample has an output value next to it's 4D coordinate. The cube has coordinates [0,0,0,0]..[1,1,1,1].
Given a new coordinate (4D) how can I come up with the best interpolated value given these samples? Eg how do I choose the samples to start with, how to interpolate.
As a first guess I would guess that this can be done with a two step process:
find the smallest convex pentachoron (4D equivalent of the 3D tetrahedron / the 2D triangle) around the coordinate we need to interpolate.
interpolate within this tetrahedron.
Especially step 1 seems quite complex and slow.
Here's the first approach I'd try.
Step 1
Find the point's 4 nearest neighbors by Euclidean distance. It's important that these 4 points are linearly independent because next they're used to create a Barycentric coordinate system. Those 4 points become the vertices of your pentachoron (aka 4-simplex).
If nearest-neighbor checks are too slow, try structuring your data into a spatial lookup tree that works in 4D.
Step 2
Now we need to associate a value with the interpolation point X. Start by deriving X's representation in this new Barycentric coordinate system. This Barycentric coordinate consists of 4 numbers, which collectively describe the relative distance between the interpolation point and each of the 4-simplex's vertices.
Normalize the Barycentric coordinate so its components sum to 1.
Each of those 4 simplex vertices are data points and have an output value. Combine those 4 output values into a vector.
Finally, interpolate by calculating the dot product of the normalized coordinate with the vector of output values.
Source: This idea is really just a 4D extension of this gem in middle of the Barycentric coordinate system page on Wikipedia.
Recently I've been planning out how I would run a game with an environment/map that is capable of unlimited dimensions (unlimited being a loose terms as there's obviously limitations on how much data can be stored in memory, etc). I've achieved this using a "grid" that contains level data stored as a String that can be converted to a 2D Array that would represent objects and their properties.
Here's an example of two objects stored as a String:
"game.doodads.Tree#200#10#terrain$game.mobiles.Player#400#400#mobiles"
The "grid" is a 3D Array, of which the contents would represent the x/y coordinate of the grid cell. The grid cells would be, say, 600x600.
An example of this "grid" Array would be as follows:
var grid:Array = [[["leveldata: 0,0"],["leveldata 0,1"]],
[["leveldata: 1,0"],["leveldata 1,1"]]];
The environment will handle loading a grid square and it's 8 surrounding squares based on a given point. ie the position of the Player. It would have a function along the lines of
function loadCells(xp:int, yp:int):void
This would also handle the unloading of the previously loaded cells that are no longer close enough to be required. In the unload process, the data at grid[x][y] would be overwritten with the new data, which is created by looping through the objects in that cell and appending each new set of data to the grid cell data.
Everything works fine in terms of when you move in a direction, cells are unloaded/saved and new cells are loaded. The problem is this:
Say this is a large city infested by zombies. If you walk three grid squares in any direction and return, everything is as you left it. I'm struggling to find a way to at least simulate all objects still moving around and doing their thing. It looks silly when you for example throw a grenade, walk away, return and the grenade still hasn't detonated.
I've considered storing a timestamp on each object when I unload the level, and when it's initialized there's a loop that runs it's "step" function amount of times. Problem here is obviously that when you come back 5 minutes later, 20 zombies are going to try and step 248932489 times and the game will crash.
Ideas?
I don't know AS3 but let me try give you some tips.
It seems you want to make a seamless world since you load / unload cells as a player moves. That's a good idea. What you should do here is deciding what data a cell should load / unload( or, even further, what data a cell should hold or handle ).
For example, the grenade should not be unloaded by the cell as it needs to be updated even after the player leaves the cell. It's NOT a good idea for a cell to manage all game objects just because they are located in the cell. Instead, the player object could handle the grenade as an owner. Or, there could be one EntityManager which handles all game entities like grenades or zombies.
With this idea, you can update your 20 zombies even when they are in an unloaded zone (their location does not matter anymore) instead of calling update() 248932489 tiems at once. However, do you really need to keep updating the zombies? Perhaps, unloading all of them and spawning new zombies with the number of the latest active zombies in the cell would work. It depends on your game design but, usually, you don't have to update entities which are not visible or far enough from the player. Hope it helps. Good luck! :D
Interesting problem. Of course, game cannot simulate unlimited environment precisely. If you left some zone for a few minutes, you don't need every step of zombies (or any actors there) to be simulated precisely. Every game has its own simplications. Maybe approximate simulation will help you. For example, if survivor was in heavily infested zone for a long time, your simulator could decide that he turned into zombie, without computing every step of the process. Or, if horde was rampant in some part of the city, this part should be damaged randomly.
I see two methods as to how you could handle this issue.
first: you have the engine keep any active cells loaded, active means there are object-initiated events involving cell-owned objects OR player-owned objects (if you have made such a distinction that is). Each time such a cell is excluded from normal unloading behaviour it is assigned a number and if memory happens to run out the cells which have the lowest number will be unloaded. Clearly, this might be the hardest method code-wise but still it might be the only one truly doing what you desire.
second: use very tiny cells and let the engine keep a narrow path loaded. The cells migth then be 100x100 instead of 600x600 and 36 of them do the work one bigger cell would do ( risk: more cluttter code-wise) then each cell your player actually traverses ( not all that have been loaded to produce a natural visibility-range ) is kept in memory including every cell that has player-owned objects in it for a limited amount of time ( i.e. 5 minutes ).
I believe you can find out how to check for these conditions upon unloading yourself and hope to have been of help to you.