Related
I'm a newbie in RL so please forgive me if I ask stupid question:)
I'm working a DQN project right now and it's very similar to the simplest snake game. The game is wrote in js and has a demo (in which snake moves randomly). But since I don't know how to write js， I can't pass the action value to the game during trainng process, what I'm doing now is generating random game image and training the dqn model instead.
What I want to ask is that: Is it possible to do in this way? Can the Q(s,r) still converge? If it's possible, is there anything I should pay attention to? and do I need the episilon parameter anymore?
Thank you very much:)
I'd definitely say no!
The problem is that the agent will only learn from random decisions and can never try if a learned action produces maybe even more reward. So everything he learns will be based on the starting frames.
Further, the agent will, in your case, never learn how to handle his size (if it grows like in snake) because he will never grow due to the bad random decisions.
Imagine a child that tries to ride a bike and you lift it off the bike as soon as it has ridden one meter. It will probably be able to ride one or even more meters straight but will never be able to do turns, etc.
I'm writing a very simple Java Game. Let me describe it briefly:
There are 4 players in a Map.
The map is a two-dimensional matrix with a value called "height"
The height between 2 nodes is the cost of that edge.
Use Dijkstra algorithm to help player navigate from a source to a destination.
Four players take turn to make a move. The total move is 8( left top, top, right top.... )
If they meets, fight for gold value, otherwise move to their target.
As they move, their strength decrease by the height difference between two nodes.
... etc
....
The problem that I'm encountering is that the source code is getting longer and complex day by day. And I think I'm using a wrong approach somehow, I feel so tired because of constantly changing the implementation. Here is my approach:
Write out all requirements.
Create all the object that I need with all getter and setter.
Create a static class to test the logic
Create unit test while putting the logic together
Add some more code, then change to code to fit the test
Write a big method that run, then break it down into smaller methods, then write unit test again.
If everything work fine, add more requirements, add more code
Then things are getting complicated because the more code I added, the complexity increases. No longer have time to write unit test because create a test case now requires too much work
Re-design, then change the implementation, go to step 1 again.
I'm come from a C++ background, and I'm only comfortable with writing 'static' libraries such as: stack, queue, link-listed, tree... Game is really a big challenging to me, especially I have to use Java. I understand the core programming is the same, so picking up Java was not really that bad. However, the time of looking up Java's API is not little. Further, the game logic is really hard to write. When this object moves, other object got affected..., so creating test for a method depends on many other methods...etc.
I really need an advice. Could anyone share some experience of how to write a game to me? I only have two weeks left for this assignment. I'm currently have 45 classes now, I feel so lost because the more I wrote the more it gets complex :( !
Best regards,
Chan Nguyen
First start thinking like a java programmer. Think as every thing in your game as an object, like the board, think about the properties and methods it has, its interfaces, how it interacts with the other objects.
If you need help getting started here is a great tutorial that guides you step by step to do a simple java game, this might put you in the right frame of mind to start programming your own. I strongly recommend you to follow the tutorial.http://www.cokeandcode.com/asteroidstutorial and to use the libraries that they used for developing the interfaz there.
I view game code architects with respect: games are complex systems with emergent properties at runtime, unusually intense interaction requirements (UI, controls) which makes a lot of OOP theory questionable value. It can be difficult to reuse game code. And, a lot of upfront planning work is wasted time.
Most game coders I know, beginning or veteran, succeed with a "just do it" iterative process. e.g.
1) write a minimal prototype. get a very basic system working, using the simplest, most obvious architecture you can think of. (my guy can run around the screen). 5 or 10 objects max.
2) add functionality (points, rules, traps, NPC behaviours, etc) and playtest, over and over. This hack on hack can makes for poorly structured code, but most coders can make it work.
3) rewrite. Programmers grit their teeth at some of the hacking they had to do in (2), and will want to throw it all out and rewrite. Resist this urge until the game is testable (as in, plyaers can sometimes enjoy it, somewhat), or a new feature would require rewriting. Then, rewrite pretty much EVERYTHING from scratch. This goes WAY faster than you'd expect, and results in solid, well-structured code.
Game coders do test, but comprehensive testing of ALL code is rare. two reasons: emergence and culture. Games have emergent properties at runtime ("yeah, but the points COULD go negative when the NPC is killed when ...."). Since games are usually for entertainment purposes, there is a culture of fast-and-loose testing. Games aren't as important as, say, missile control code.
I expect others with more coding experience answer this. (I have written a fair bit of code but I tend toward quick and dirty script type coding style - I know lots of coders who are way better than me.)
I'm interested in writing certain software that uses machine learning, and performs certain actions based on external data.
However I've run into problem (that was always interesting to me) -
how is it possible to write machine learning software that issues orders or sequences of orders?
The problem is that as I understand it, neural network gets bunch on inputs, and "recalls" output based on results of previous trainings. Instantly (well, more or less). So I'm not sure how "issuing orders" could fit into that system, especially when actions performed by system affect the system with certain delay. I'm also a bit unsure how is it possible to train this thing.
Examples of such system:
1. First person shooter enemy controller. As I understand it, it is possible to implement neural network controller for the bot that will switch bot behavior strategies(well, assign priorities to them) based on some inputs (probably something like health, ammo, etc). But I don't see a way to make higher-order controller, that could issue sequence of commands like "go there, then turn left". Also, bot's actions will affect variables that control bot's behavior. I.e. shooting reduces ammo, falling from heights reduces health, etc.
2. Automated market trader. It is certainly possible to make system that will try to predict the next market price of something. However, I don't see how is it possible to make system that would issue order to buy something, watch the trend, then sell it back to gain profit/cover up losses.
3. Car driver. Again, (as I understand it) it is possible to make system that will maintain desired movement vector based on position/velocity/torque data and results of previous training. However I don't see a way to make such system (learn to) perform sequence of actions.
I.e. as I understood it, neural net is technically a matrix - you give it input, it produces output. But what about generating sequences of actions that could change environment program operates in?
If such tasks are not entirely suitable for neural networks, what else could be used?
P.S. I understand that the question isn't exactly clear, and I suspect that I'm missing some knowledge. So I'll appreciate some pointers (i.e. books/resources to read, etc).
You could try to connect the output neurons to controllers directly, e.g. moving forward, turning, or shooting in the ego shooter, or buying orders for the trader. However, I think that the best results are gained nowadays when you let the neural net solve one rather specific subproblem, and then let a "normal" program interpret its answer. For example, you could let the neural net construct a map overlay of "where do I want to be", which the bot then translates into movements. The neural network for the trader could produce a "how much do I want which paper", which the bot then translates into buying or selling orders.
The decision which subproblem should be solved by a neural network is a very central one for its design. The important thing is that good solutions can be taught to the neural network.
Edit: Expanding this in the examples: When the ego shooter bot gets shot, it should not have wanted to be there; when it gets to shoot someone else, it should have wanted to be there more. When the trader loses money from a paper, it should have wanted it less before; if it gains, it should have wanted it more. These things can be taught.
The problem you are describing is known as Reinforcement Learning. Reinforcement learning is essentially a machine learning algorithm (such as a neural network) coupled with a controller. It has been used for all of the applications you mention, even to drive real cars.
Like most developers, I'm a business developer, which in essence consists of slapping a UI onto some back-end data store. (We all know there's a lot more to it than that, but that's usually what it boils down to.)
I understand that game development is very different from business development, but I'm having a hard time explaining it to a friend of mine. I was hoping the SO community could help me out here.
To me, modern game developers deal a lot with manipulating 3-dimensional graphics. In gaming code (and I'm guessing here), you're assembling polygons (or something like that), rotating 'em, etc. This involves a different way of thinking from manipulating relational data (for instance). I don't know, really. I just know it's different.
EDIT:
I should stress that by "development" I mean "programming," not all of the aspects that go into creating a game or piece of business software. I'm sorry I didn't make that clear originally.
Thanks!
I'm in game development but came from business development long ago. Game development is very rigorous in mathematics if you work on the physics or graphics side. Even AI can need quite a bit of mathematics for the low-level stuff. The hardware usually takes care of a lot of the polygon manipulation math as far as drawing on the screen goes. There is also a lot of involvement with generating the in-game data with (often) many tools that are run in a pre-processing step, and that too can be math-intensive if you are generating visibility data.
In terms of programming domains, amongst other things, we deal with:
Graphics programming (including shader development)
Animation
Physics simulation
AI and gameplay
Audio
Networking (typically fairly low-level stuff)
Some of these involve pretty serious maths and algorithms knowledge. On top of all that, we face extremely tough speed constraints, and typically have to be very careful with memory usage too. We face constantly changing hardware, and since we're trying to push hardware to the limit, this can be pretty tough - you can't just abstract it away. Most game development is still quite low-level C++ work. We probably deal with databases less than most other programmers nowadays (although online games are changing this)!
Programmers are often the minority on modern game projects: it's all about content creation (animation, modelling, texturing, audio and design). This means many game programmers are dedicated to making the content creation process efficient, rather than working on the game code itself. This work may have more relaxed speed and memory constraints, although it does have to deal with massive data sets.
Making the game 'fun' is one of the hardest things to do - in business terminology, it "means extremely unstable requirements" as the designers constantly change their mind about how things should work, to chase down that elusive fun factor.
Finally, games are generally a ship-once, no chance to fix stuff kind of deal. This actually means there's very little code maintenance involved, so traditionally there may have been less attention paid to code quality issues. This is changing now with the growth in post-launch content addition, online gaming and the sheer size of modern projects.
Overall it's an incredibly exciting field to be in, the downside is that it's often less well paid (because it's a very tough business financially for developers, and because it's popular, there's always a fresh supply of people looking for jobs).
Just some random thoughts about what is different in game development. Note that there might be some sarcasm in it, though I tried to suppress the urge.
Unless you're a lucky employee of one of those new-style studios (like Eidos Montreal or Blizzard), there is always a deadline to fear that is much too short. In business programming, you mostly make the deadline up for yourself.
A business application serves some specific need. A game's intent is to entertain people. You can't really predict if a game will fail until it's out.
Performance is essential, in every aspect of the game. Writing code that is good to maintain is second priority. In business programming, good code that works is top priority.
For a business application, a shiny UI is a bonus. For a game, it is a must.
Debugging games is much harder, because there is always some hardware dependence which results in bugs that can only be reproduced on some machines, none of which is in your company. And a game sucks up much more performance than a typical business application.
You have people dedicated to creating the art, story, music, sound, background and design, none of which necessarily needs programming knowledge (scripting is a little different), i.e. you have a lot of content which is what the users (players) will see. Nobody cares about how good your code is, unless performance is bad or there are bugs. The others get the praise.
For larger games, you have programmers dedicated just to 3D graphics, networking, audio, tools, scripting, physics and so on. Most of them are highly specialized and each of them can lead the game into a disaster. You'll only need advanced math skills if you're the graphics or physics guy. Well, or AI.
Most games are fire-and-forget, apart from some bugfixes, unless it's one of the more successful games, which get an expansion pack or a sequel.
Security is an important issue for online games, since there are much more annoying people trying to to put people off than there are for business applications, many of which are for (more or less) internal uses at the customer.
You are expected to work much more than when writing business applications.
To land a job for an AAA title, you need to have worked on at least three shipped AAA titles (no, no typo here, ever read some job descriptions at Blizzard or LucasArts? :P)
But here come the good things:
You can pretend to work when you're playing games.
And finally, programming games is fun. Priceless.
Business development is generally much more forgiving.
The reason is basically this; usually, people ARE PAID to use business software. People PAY to use game software.
This may sound like it's not answering your question, but it really is. When my boss says "use microsoft word for that document", they're providing the software, and I'm obligated to use micosoft word. And so, when using it, when it decides to renumber all my chapter headings "just because" or a save to disk takes 30 seconds while it resolves OLE references (it's JUST ONE FREAKING EXCEL SPREADSHEET, for heaven's sake!), I just grit my teeth and remind myself I'm getting paid to do this.
Whereas, if I'm in a game, I'm expecting entertainment. I'm expecting the experience to work properly, and smoothly, and cleanly, with no major stutters or problems.
Again, getting down to why this is an issue for programming; those loops and structures in the game had better be DAMN good to make sure there is no major slowdown, no stuttering in the game engine, nothing that makes the consumer who just spent X amount of his hard-earned dollars say "this is a piece of crap" and walk away. With business software, you can get away with that sort of thing; in some ways, it's almost expected. Again, look at the performance of Microsoft Word; if it were a game, it would be laughed out of existence.
I know I sound like I'm picking on Microsoft Word, and I generally am, because I find it to be hideous, but the point is true for so many pieces of software. CAD software is another example. Same basic things going on as in games, but in general it's slow and hard to work with without a lot of training.
The difference comes down to polish, and the level of polish that's expected. Yes, there's generally more flexibility in business software than there is in games; but moreover and more importantly from a coding perspective, the code has GOT to work efficiently and cleanly in a game; business software is, generally, more forgiving of sloppy code.
In a business app, unoptimized and slow algorithms are generally accepted; and while they're never preferable, frequently the business decision gets made to add another feature instead of improving the performance. But in games, performance IS a feature, and one which is make-or-break.
One should have infinite loops, one shouldn't.
One should have infinite loops, one shouldn't. - Rich Bradshaw
Rich is right. Fundamentally, from a coding standpoint, a game loop creates a "frame" of action in which actions are taken based on the state of the game such as controller input, object collisions, etc. This loop repeats infinitely until some state of some game element or input tells it to stop or "quit." This approach keeps the CPU and graphics card pretty busy, hence the market for gamer machines with fast processors and even faster graphics cards.
Business applications do not have an active loop. Instead, they sit idle waiting for an event such as a click, a message from a web service client, an HTTP GET request, etc. Then they respond to the event.
Sure, gaming is generally more geometrically intensive than business applications, but that is not entirely true. Consider image editing, CAD and graphics tools. For many, these are business applications. But for the most part, a business application has to do with querying data, displaying that data, accepting user input, and modifying the data based on user input. In many cases, the business application does this across the network or even the Internet, but it's an apt nutshell.
The skillset and mindset of a business application developer and the game developer is often different. The game developer has a limited number of input constructs to consider in terms of creating a user experience with an unlimited choice of context or "world" if you will. The business developer is the opposite, with a limited set of potential contexts, usually the web page or the basic window, and an unlimited (or nearly so) set of input and data display combinations to create a user experience entirely different than the game developer sets out to achieve.
One big difference between business development and game development is the number of disciplines involved. Most business software is created by a team of developers, who all have the same basic skillset. In contrast, a game is created by a team of game designers, visual artists, 3d modelers, animators, musicians, and developers.
Good points about mathematics and integration of artists and other specialists in the team. In addition, I'd say that:
Game development, to some extend, will be more hardware dependent. In many cases, games are built simultaneously to several platforms and consoles (not to mention cellphones), with different architectures. That is abstracted up to a certain extent, but developers cannot completely avoid this fact.
Game development is often more performance sensitive, or at least the performance requirements are different. You're dealing with real-time experience, so a lot of time is spent optimizing those pesky fps.
In many cases, game development does not care as much about reuse and maintainability. The game engine will probably be reused, but the application code base will probably not live to see v2.0. In the last stretch of a project, there is a lot of quick and dirty debugging going on. If it looks fine to the end user, there's no added value in making an elegant fix two days before the release.
Let's start from the goal - the goal of game development is to create entertaining product. It should be accurate to the extend that it looks good and runs smoothly. The goal of a business software solution is to model a work process. It should be a tool which works fast enough. A stable product, which executes absolutely accurately and secure the tasks it should do.
Since we target different goals, we use different approaches to build a game and a business software. Let's move to the requirements. For a game, the requirements are determined by the game designer. For a software product the business defines the process and the requirements. For a game the requirements are not final - shall we have small cartoon figures or real human models - this does not matter for the game engine for example. But for a software product, the requirements should be strictly followed and cleared to the maximum possible detail before development.
From the different requirements come different software design and development approach. For a game the performance and gameplay are critical and the qualiity of the graphics and sounds (for example) could be reduced just to be compatible with weaker hardware. Also the physical model could be simplified just to run smoother and improve the gameplay. For the business software everything should be exact and cutting features means that your product will not be as functional as designed anymore.
For a game, the security is not important - there is no critical customer data which should be saved. For a business software a good security system should be supplied - starting from data encryption (while saving on data storage or transferring through network) moving through backup system and mentioning (but not last) the compatibility with previous versions.
I could continue with other aspects but I guess this is already too much for one post...
Business software (that isn't shrink-wrap software) can generally be much more poorly written but still considered a commercial success due to the bizarre disconnect between the quality of the product and saleability of the product. Game software, on the other hand, has to actually be good to survive the marketplace.
The bar for quality in specialized business software is generally much lower.
Business software has to be reliable, maintainable, consistent, not be too annoyingly slow, and can build on lots of already written stuff, such as databases, controls, forms etc.
A games programmer often starts with a blank sheet - hardware reference manuals, some documentation about the hardware and usually thin vendor libraries around some advanced hardware that's completely different to the last job.
From this they have to build what you see - and make most of it work within a 20ms time period, reliably, and often within a ridiculously short time period, facing changing requirements and often a very hard deadline, working untold numbers of hours for a comparative pittance.
That's not to mention often having to master some fairly complex mathematics and physics....
Performance is really the difference, from what I can tell.
Technologywise, games are usually Windows/C++ driven.
Game programming has more in common with scientific programming. You are modeling behavioral systems and anticipating results based upon a limited set of input.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
Courses for people who are being introduced to programming very often include a code project, which I think is a nice way to learn. However, such projects often feel too artificial, and are thus not very rewarding to work on.
What are your ideas of rewarding code projects? (Preferably easy to begin, and extendable at will for the more advanced!).
Edit:
#Mark: thanks for the link, though I'm more interested in projects for people who are completely new to programming (the link seems to refer more to people who are already proficient in at least one language, and trying to learn a new one -the typical SO audience I'd say :) -).
#Kevin, Vaibhav, gary: I was thinking of people who are learning programming through one language, so at the beginning of the course some don't know anything about control structures (and even less about any kind of syntax). However, I was thinking in quite a large project (typically in the 1k-10k lines of code range, possibly in groups of 2 or 3 students). This is what was done at my school for the complete beginners, and it sure seemed to work for them... except that most of them found their projects quite boring to work on!
As has been stated a few times, what you are trying to teach the beginner is very important to the project.
My advice to you for planning something like this:
1) Avoid making a computer game
A computer game, while fun to build, doesn't reward the programmer with results early on (it's very complex). You want to concentrate on small but useful application programs, such as a Port Scanner. The example there is a little complex, but it's one of the best learning projects I've seen on the web.
2) Teach graphics early
It's rewarding to see the fruits of your labors early on, and it motivates you to go further. Whether you're using WinForms, MFC or the Win32 API, OpenGL or DirectX, teach it early.
3) Many small lessons with in depth information
This principle is followed by the above linked Port Scanner project, and it works well. Teach each part thoroughly, and give time for the beginner to absorb the lesson. I think that ZophusX had a good format for giving the information. It's too bad he's mostly abandoned his site.
4) It takes time
Don't rush things. Nobody becomes a stellar programmer in a few weeks. Try and make the lessons simple, but engaging, and keep building from your previous lessons.
5) Get feedback early and often
You might think a project is incredibly interesting, or a particular lesson or such, but you aren't the one learning. Your student(s) will greatly appreciate it when you ask them early on how things are going, and what they'd like to know more about. Be flexible enough that you can accomodate some of those requests.
6) Have fun teaching
Have fun. Passion is contagious, and if your student(s) see how much you enjoy the subject matter, some of that enthusiasm will rub off on them as well.
I hope that helps!
Some good rewarding projects, in terms of what you can learn and which are quite scalable in terms of complexity, features are:
Games
A travel and transportation reservation/booking system
Encyclopedia or a Dictionary of terms, articles
Conversion Calculators (Currency, Units, etc.)
The key is to pick a project simple enough, so that some of its features are immediately apparent, when you look at the project title. And when really given a thought, will reveal more features that you can add to it.
The project should have enough difficulty to so that its features seem just beyond the beginner's reach, thereby motivating him to learn something new all the time.
If you are training new people in your company, then attaching them as intern resources on a live project is very rewarding.
This increases the work load of the main developers a little (because they have to review all the work that the intern does), but goes a long way in terms of training and development of the person.
I do think that games and puzzles are a good place to start as they can give great scope for developing more complex versions. For example a tic-tac-toe program can be built as a simple command line program initially that lets two players play the game.
This step can be used to show how a simple data structure or array can represent the game board, simple input to get user commands/moves, simple output to display the game board and prompts etc. Then you can start showing how an algorithm can be used to allow player vs computer mode. I like the simple magic square math algorithm for tic-tac-toe as it's based on very simple math. After this the sky's the limit, UI improvements, using file I/O to load and save games, more advanced algorithms to get the computer to play better etc. More complex and satisfying games can still be produced using text mode or simple graphics.
I've used the Sokoban game as a means of showing lots of techniques over the years.
The simplest game I've used is a number list reversing game. This involves a mixed up list of numbers from 1-9. The player can specify a number of digits to reverse on the left of the list. The aim is to get the list sorted. This is great for absolute beginners. Each little part of the game can be written and tested separately.
It really depends on what you're trying to teach the beginner. If you're trying to teach syntax, then simple "Hello World" programs and ones that spit out every odd number between 1 and 100 are fine to get them started. If you're trying to teach data structures, then maybe something like a 20 questions game or some simple sorting program. If you're trying to teach recursion, then maybe a breadth first search program. If you're trying to teach database manipulation, then something like a order tracking system would be appropriate.
Take a look at code examples in the book Python Programming for the Absolute Beginner
Text Adventure.
It's a console app
You'll need to do some useful things, hold inventory, map and room state and parse input
It's fun and you can give it to others to play! :D