I'm a newbie in RL so please forgive me if I ask stupid question:)
I'm working a DQN project right now and it's very similar to the simplest snake game. The game is wrote in js and has a demo (in which snake moves randomly). But since I don't know how to write js， I can't pass the action value to the game during trainng process, what I'm doing now is generating random game image and training the dqn model instead.
What I want to ask is that: Is it possible to do in this way? Can the Q(s,r) still converge? If it's possible, is there anything I should pay attention to? and do I need the episilon parameter anymore?
Thank you very much:)
I'd definitely say no!
The problem is that the agent will only learn from random decisions and can never try if a learned action produces maybe even more reward. So everything he learns will be based on the starting frames.
Further, the agent will, in your case, never learn how to handle his size (if it grows like in snake) because he will never grow due to the bad random decisions.
Imagine a child that tries to ride a bike and you lift it off the bike as soon as it has ridden one meter. It will probably be able to ride one or even more meters straight but will never be able to do turns, etc.
hoping for some advice on my architecture
Currently I used a layered architecture but somethings are getting a bit complex and was thinking a message bus would be far more useful
I have a controller action called "CreateTeam" on the TeamControler
Which fires of the to TeamService method CreateTeam and then also fires of to the PlayerService method "CreatePlayers".
Which means my TeamController method "CreateTeam" has two responsilibites
But when the user creates a team, I need players to be created also.
So would it be better that CreateTeam, also fires an event TeamCreated which was picked up elsewhere? Giving single responsibility and separating concerns?
Also, I have never used Message bus pattern before so hope im not confusing anyone but is it right that events are raised which the bus picks up and anything listening then gets? So this would be Tell don't ask?
And finally, the listeners on the message bus, could they be web api controllers? Rather then lots of seperate applications? So my PlayerController method with CreatePlayers, could be fired of when CreateTeamEvent is raised?
Somewhere in your system you have to have the knowledge (responsibility) embedded to call your 2 separate functions. The Servicebus could be used to do this but in my experience that would be an overly complex and more difficult to understand resolution.
I would stick with a layer (service class) as suggested by #S. Baggy.
I made a flash game with ActionScript 3. I want when the character falls into water or swamp
or his score=0, the game to be start from first.
I want load scene from first.
I am a beginner in as3. Please help me by details.
But I do not know what code to write.
I'm grateful for your advice.
I can only suggest grouping your level dependent variables and writing your own reset() method that is used to initialise and start the game. Then calling that whenever one of the loss conditions are met to restart the game.
As far as I know, this may not always be a quick and clean thing to do, nor do I know of any single "one size fits all" solution that won't causing any further problems down the line. This is more to do with how your design and layout your game code.
As far as I know, it's really not possible, but I just want to be sure before I'm moving to flash.
can I make an html5 game secure enough so people won't be able to change their score and other variables while playing?
thanks!
There is no "depends", the straight answer to your question is "no" and I think my fellow answerers simply muddied the waters.
You cannot trust the client. With any language, whether you're writing assembly or HTML or Flash, you cannot trust the client. No matter how much you wrap your code in obfuscation and such, it can and will be figured out (and often quicker than you might think).
This is stressed everywhere and yet people keep getting bit by it. Online games get "speedhacked" because they don't check the velocity of players, or they get item duplication because they don't verify that a player actually has an item that they're trying to do something with, or the lame little flash games get hiscore entries of 9999999 because a simple tool like Tamper Data (a Firefox add-on) is all it takes to change the score as it's sent to the server.
You can't trust the client, whether HTML5 or Flash.
If it's a single-player game, let the player cheat. That is their decision. If it's a multiplayer game, the server verifies every step of the game and anything outside of the rules is thrown out. If it's hiscores, send a replay of the game to the server and analyze it for any cheating rather than sending just a numeric score.
since your users can see all the source code this is a rather complex problem.
they can easily change any function or variable at runtime without your script ever knowing.
even if use a complicated signing function to validate the results.
and i am sorry but i don't think colins way would work either. i could just change any input to make the server do whatever i want.
maybe a constant monitoring of the score thru the server would be able to detect any impossible changes. still someone cheating in the realms of "possible" results would be uncaught.
in the end i would say u can only make it rather difficult to cheat but not impossible for someone with a little bit of skill.
don't use it for any games where u can win something by scoring the highest.
since the matter seems rather puzzling to people:
flash delivers compiled swf files, that cannot (since flash 9) be decompiled to useful.smth
so u can put a secret in there which you use to sign the score.
i.e. send the score and the md5 of score+secretkey. so the server (which also knows the key, can check it).
furthermore flash variables are not so easy to temper with (you would have to find them in ram and alter them there, which is a very complex task), while javascript vars can be easily edited using, for example, webkit developer tools
update
actually i correct myself => all swfs can be decompiled
this just leaves us with code obfuscating and "encrypting"
i guess the world is a bad place after all ;)
Depends on the way your game is coded, but if all the logic is sent to the client and only the score returned then you have no hope. Only by returning the inputs and calulating the score on the server side can you try to prevent the users submitting any score they wish.
Don't forget, by definition the user must change their score or it could never be more than 0...
One Thought!!
You may use Knockout.js to modify your score and other variables as observable properties.
The steps are:
Create ViewModal for your game
Create observable properties for all the variables (i.e score)
You need to store the score in cache so that you can access it when new score arrives.
Attach custom subscriber to these properties and write logic to check the score should be updated by a "UNIT" at a time ( by unit I mean, how you suppose to update user's score at a time). The difference between the last score and current score should not go beyond the "UNIT"
update scroe as ViewModal.Score(newScore); //this would fire an event to the subscriber of observable property.!
Over the course of time, my team has created a central class that handles an agglomeration of responsibilities and runs to over 8,000 lines, all of it hand-written, not auto-generated.
The mandate has come down. We need to refactor the monster class. The biggest part of the plan is to define categories of functionality into their own classes with a has-a relationship with the monster class.
That means that a lot of references that currently read like this:
var monster = new orMonster();
var timeToOpen = monster.OpeningTime.Subtract(DateTime.Now);
will soon read like this:
var monster = new Monster();
var timeToOpen = monster.TimeKeeper.OpeningTime.Subtract(DateTime.Now);
The question is: How on Earth do we coordinate such a change? References to "orMonster" litter every single business class. Some methods are called in literally thousands of places in the code. It's guaranteed that, any time we make such a chance, someone else (probably multiple someone elses) on the team will have code checked out that calls the .OpeningTime property
How do you coordinate such a large scale change without productivity grinding to a halt?
You should make the old method call the new method. Then over time change the references to the old method to call the new method instead. Once all the client references are changed, you can delete the old method.
For more information, see Move Method in Martin Fowler's classic, Refactoring.
One thing you can do is to temporarily leave proxy methods in the monster class that will delegate to the new method. After a week or so, once you are sure all code is using the new method, then you can safely remove the proxy.
I've handled this before by going ahead and refactoring the code, but then adding methods that match the old signature that forward the calls to the new method. If you add the "Obsolete" attribute to these temporary methods, your code will still build with both the old method calls and the new method calls. Then over time you can go back through and upgrade the code that is calling the old method. The difference here is that you'll get "Warnings" during the build to help you find all of the code that needs upgrading.
I'm not sure what language you're using but in .Net you can create compiler warnings which will allow you to leave the old references for a time so that they will function as expected but place a warning for your other developers to see.
http://dotnettipoftheday.org/tips/ObsoleteAttribute.aspx
Develop your changes in a branch. Break out a subset of code to a new class, make changes across the client base, test thoroughly, and then merge back.
That concentrates the breakage to when you merge — not the entire development cycle.
Combine this with Patrick's suggestion to have the monster call the small monsters. That'll let you easily revert if your merged client code breaks changes to that client. As Patrick says, you'll be able to remove the monster's methods (now stubs) once you prove nobody's using it.
I also echo several posters' advice to expose the broken out classes directly — not via the monster. Why apply only half a cure? With the same effort, you could apply a complete cure.
Finally: write unit tests. Write lots of unit tests. Oh, boy, do you need unit tests to safely pull this one off. Did I mention you need unit tests?
Keep the old method in place and forward to the new method (as others have said) but also send a log message in the forwarding method to remind yourself to remove it.
You could just add a comment but that's too easy to miss.
Suggest using a tool such as nDepend to identify all of the references to the class methods. The output from nDepend can be used to give you a better idea about how to group the methods.
var monster = new Monster();
var timeToOpen = monster.TimeKeeper.OpeningTime.Subtract(DateTime.Now);
I'm not sure that divvying it up and just making portions of it publically available is any better. That's violating the law of demeter and can lead to NullReference pain.
I'd suggest exposing timekeeper to people without involving the monster.
If anything you'd be well off analysing the API and seeing what you can cut and encapsulate within monster. Certainly giving monster toys to play with as opposed to making monster do all of the work itself is a good call. The main effort is defining the toys monster needs to simplify his work.
Don't refactor it.
Start over and follow the law of demeter. Create a second monster class and start from scratch. When the second monster class is finished and working, then replace occurrences of the first monster. Swap it out. Hopefully they share an interface, or you can make that happen.
And instead of this: "monster.TimeKeeper.OpeningTime.Subtract(DateTime.Now)"
Do this: monster.SubtractOpeningTime(DateTime.Now). Don't kill yourself with dot-notation (hence the demeter)
Several people have provided good answers regarding the orchestration of the refactor itself. That's key. But you also asked about coordinating the changes between multiple people (which I think was the crux of your question). What source control are you using? Anything like CVS, SVN, etc can handle incoming changes from multiple developers at once. The key to making it go smoothly is that each person must make their commits granular and atomic, and each developer should pull other people's commits often.
I will look at first using partial class to split the single monster class over many files, grouping methods into categories.
You will need to stop anyone editing the monster class while you split in into the files.
From then on you are likely to get less merge conflicts as there will be less edits to each file. You can then change each method in the monster class, (one method per checkin) to call your new classes.
Such a huge class is really an issue. Since it grew so big and nobody felt uncomfortable, there must be something wrong with project policies. I'd say you should split into pairs and do pair programming. Create a branch for every pair of programmers. Work for 1-2 days on refactoring. Compare your results. This will help you avoid the situation when the refactoring would go from the start into the wrong direction and finally that would lead to the need of rewriting the monster class from scratch.