I have been trying out kubernetes on GCP to build microservices for a while and it has been amazing so far.
Although i am a bit confused on what would probably be the best approach, should i
create (gcloud container clusters create "[cluster-name]" ...) one container-cluster per service?
create one container-cluster for multiple services? or
do both of those above depending on my situation?
all of the examples i have managed to find has only covered #2, but my hunch is kind of telling me that i should do #1, and my hunch is also kind of telling me that i have probably missed some basic understanding around containers, i have been trying to find answers without any luck, i guess i just can't figure out the right search keyword, i am hoping that i could find some answer here.
I suspect the answer is "it depends" (Option 3)
How isolated do you need each application to be? How much redundancy (tolerance of VM failure) do you need? How many developers will have access to the Kubernetes cluster? How much traffic do the apps generate?
If in doubt I recommend running all your apps on a single cluster. Simpler to manage and the overhead in providing highly available infrastructure is then shared. You'll also have greater levels of VM utilization, which perhaps might result in reduced hosting costs. The latter is a subjective observation, some apps have very little or occasional traffic resulting in very bored servers :-)
Different kind of software offer different amount of configuration/customization. Routers are one of the most configurable software systems I know of. I want to know how routers handle configurations - how they alter the code flow based on the configuration?
One obvious way is to use if..else clauses provided by most of the language(let's assume we are using C)
So is there any other programming method(or paradigm?)
Data-driven programming paradigm may be viable one. Configuration can be thought of one of the input source and so can be used to alter the code flow.
What I need to know is, is there any papers and references that I can use to enrich my understanding. Not just routers any kind of software. If the question seems to vague, let me know I will add more details.
I don't know anything about configuration of routers, but your question states you are interested in configuration for any kind of software, so the following might be of interest to you.
I am the author of Config4*, which provides C++ and Java parsers for a particular configuration syntax. I suggest you do the following. Skim Chapters 2 and 3 of the "Config4* Getting Started Guide" (HTML, PDF) to get an overview of the configuration syntax and API. Then take your time reading the "Config4* Practical Usage Guide" (HTML, PDF), which discusses the "how to" for a variety of different ways to use configuration. Although the discussion in that manual makes use of the Config4* syntax and API, the principles could be used with another syntax, for example, XML. If you focus on the principles discussed in that manual, rather than the syntax, then I suspect you will start to develop some insight into how a router handles its configuration.
What reasons are there against (or for) using the features of an Enterprise Service Bus when building an overall service adhering to a micro-service architecture (http://martinfowler.com/articles/microservices.html)? Why should we use dumb pipes and smart endpoints as opposed to using smarter pipes and be able to develop simpler services?
This is a huge question and probably can't be answered effectively in SO's Q&A format.
It depends what you are doing with it.
If you are building a single product which consists of lots of small pieces of function that can be thought of as being independent then microservices maybe the way to go.
If you are a large enterprise organisation where IT is not the main consideration of the board of directors as a competitive advantage and you work in a heavily regulate industry where new standards have to be applied across globally distributed projects with their own IT departments, some from new acquisitions, where you can't centrally control all the endpoints and applications within your organisation, then maybe you need an ESB.
I don't want to be accused of trying to list ALL the advantages of both approaches here as they wouldn't be complete and may be out of date quickly.
Having said that, in an effort to be useful to the OP:
If you look up how Spotify and Netflix do microservices you can find many things they like about the approach, including but not limited to: ease of blue/green deployment of individual services, decoupled team structures, and isolation of failures.
ESBs allow you to centrally administer and enforce policies, like legal requirements, audit everything in one place rather than hoping each team got the memo about logging everything, provide global statistics about load and uptime, as well as many other things. ESBs grew out of large enterprises where the driver was not customer response time on a website and speed of innovation (amongst other things) but Service Level Agreements, cost effectiveness and regulations (amongst other things).
There is a lot of value in both approaches. Microservices are being written about a lot at the moment, just as ESBs were 10-15 years ago. Maybe that's a progression, maybe it's just a change, maybe it's just that consumer product companies need to market themselves and large enterprises like to keep details private. We may find out in another 10 years. For now, it depends heavily on what you are doing. As with most things in programming, I'd start out simple and only move to the more complex solution if you need to.
The term ESB has gotten overloaded, primarily in the Java world, to mean a big and complex piece of infrastructure that ends up hosting a bunch of poorly implemented logic in a central place.
Lighter-weight technologies like Apache Caml or NServiceBus don't encourage this kind of approach and indeed follow the "dumb pipes / smart endpoints" approach that has served as the backbone of the internet from the beginning.
NServiceBus specifically focuses on providing a higher level framework than most messaging libraries to make it easier to build smart endpoints that are more reliable through its deeper support for once-and-only-once message processing.
Full disclosure - I'm the founder of NServiceBus.
Because services are isolated and pipes are reused.
Core idea of microservices is isolation - any part of the system can be replaced without affecting other services. Smart pipes means they have configuration, they have state, they have complex (which often means hard-to-predict) behavior. Thus, smart pipes are less likely to retain their exact behavior over time.
But - pipe change will affect every service attached while service change affects only other services that use it.
The problem with how ESB is used is that it creates a coupling between ESB and services by having some business logic built into the ESB. This would makes it more difficult to deploy a single service independently and increasingly making the ESB more complex and difficult to maintain.
I've started using Raven for my last project. When my boss learned about it, he mentioned it's based on Access and he had very bad experience with multiple users and Access. Now I have to either switch or prove to him he is wrong.
No, it isn't. The confusion is because RavenDB can use ESENT for data storage and ESENT used to be called Jet Blue. It was called Jet Blue because it was originally developed to replace the Jet Red engine which was/is used in Access. The Wikipedia entry is quite accurate about the history and differences.
Laurion's answer is correct, but I also wanted to point out that in Raven you can swap out the ESENT storage engine for another that Oren developed called Munin.
From Ayende's blog post about Munin.
Raven.Munin is the actual implementation of a low level managed storage for RavenDB. I split it out of the RavenDB project because I intend to make use of it in additional projects.
At its core, Munin provides high performance transactional, non relational, data store written completely in managed code. The main point in writing it was to support the managed storage in RavenDB, but it is going to be used for Raven MQ as well, and probably a bunch of other stuff as well. I’ll post about Raven MQ in the future, so don’t bother asking about it.
Munin is a low level api, not something that you are likely to use directly. And it was explicitly modeled to give me an interface similar in capability to what Esent gives me, but in purely managed code.
Later this year I want to release a PHP framework that I've been working on as open source. I do use source control (SVN), but it's on an extremely limited basis. I'm self-taught, I develop by myself and don't have the experience of working with large teams. I have some ideas about what can help make a project successful, but I'm fuzzy on some of the details. Since it's not yet released, I want to do everything I can to set up the right infrastructure from the beginning. What do I need to know in order to setup and manage a successful project?
Some ideas that I have to make it successful (beyond marketing it):
Good documentation and tutorials
Automated unit tests and builds to
push update to the website
A clear roadmap
Bug Tracking integrated with the
source control
A style guide to keep the code
consistent
A forum for the community to get
support, share ideas, etc.
A good example application built with
the framework
A blog to keep the community informed
Maintaining backwards compatibility
wherever possible
Some of my questions:
How do I setup and automate a one
step submit-test-commit-generate API
docs-push update to website process? Edit: Would Ant or Maven be good candidates for this? If so, do you know of any resources for setting up a PHP project using them?
How do I handle (technically)
submissions from other users? How can
I ensure that those submissions must
be approved before being integrated?
What are some of the pitfalls that
can be avoided in terms of the
project community? I'd prefer to have
it be as friendly and helpful as
possible without a lot of drama.
I'd love to learn from your experience on any of these points. If you think I'm missing anything big, please share that as well. Any resources (preferably geared toward a beginner) that you could point me towards would also be greatly appreciated.
I'm just getting started in community projects, but I'll give you some advice on what I know.
How do I setup and automate a one step submit-test-commit-generate API docs-push update to website process?
I've never implemented it as one process. You could just have a checklist, and possibly even create some scripts to do certain tasks. I've never worked with any source control that automates the uploading and such to be done by a script. Most of the time, there is some web interaction to be involved.
You don't want to push API changes until it's an official release.
EDIT: Working Environment
For PHP, most of the time, I either edit directly on the server and test it there, using a beta.example.com, or similar, before pushing to example.com. You could also set up an web environment on your home PC (using XAMPP for Windows, or the standard LAMP installation on Linux). You would probably just use a mirror of your repository here, so you'd do svn commit, or whichever is appropriate for the VCS or DVCS you choose.
The fun part is testing this with different PHP versions. I've not done this myself, but you could probably use a .htaccess file to run a different PHP binary in order to test it out. I'm not really sure what the best option is for this is.
I've not done much with API, as I've never created a library, but just doing a quick search I found http://www.phpdoc.org/. It looks like a mature project, so that might be a starting point.
As far as creating releases go, I generally create a script that only includes the files that are part of the distribution (it will filter out any VCS files, and anything that you don't want in the distributed file). You could write a script around find on linux (which is what I do most of the time), or there may be other better options.
How do I handle (technically) submissions from other users? How can I ensure that those submissions must be approved before being integrated?
This is mostly handled by the bug tracker, and limited access in the Version Control System. Usually, you, and the people you allow, can commit to the VCS. Other users can submit patches, but then you might have someone review the patch, test the patch, and commit. You could split these tasks up as a team, or assign a patch to one person and have them do it all.
What are some of the pitfalls that can be avoided in terms of the project community? I'd prefer to have it be as friendly and helpful as possible without a lot of drama.
I would just make sure to keep it as positive as possible with the project members and community. There's going to be some disagreements, and it will drive a few people away, but as long as you have a stable product that meets the needs of most people, I think that's all that anyone can expect.
One minor suggestion that's worked well for me: start using first-person plural pronouns, rather than singular ones. That is, talk about "we" and "us" rather than "I" and "me." It encourages other people to participate when they feel like part of team, rather than when they feel like they're contributing your own self-aggrandizement.
The most important thing you have to do is to attract users. Without users, you won't get any contributions and developers helping you out. Because developers are users first, and then they decide to extend/fix something they use and might become contributors.
So to get users, you should consider
describe what your framework does in one or two sentences at the top of your project page
mention how your framework can be used and for what, what situations it is most useful for
add a lot of examples on how to use it
mention whether your framework is stable, beta or alpha. That's important because user need to know that before they start using it
also mention whether you want to keep improving it and keep working on it - most users don't want to use a framework that's abandoned (also keep in mind that a lot of users check your commits to see whether you really are working on it - if your last commit to the repository was months ago then you're not really working on it, so cheating isn't possible)
If you got all this, and people start submitting patches, you can use a patch tool to apply those to your source. Depending on your version control system, you can either use the GNU patch, a diff/patch tool that comes with your version control or maybe even a GUI tool that helps you with this. SVN doesn't have a patch tool (yet), but 'svn diff' will create a patchfile which you can then apply with the GNU patch tool, or in case you're using TortoiseSVN, right-drag the patchfile to your working copy and have TortoiseMerge apply it for you.
And on how to best deal with the community:
answer questions in time, don't wait more than two or three days to answer questions
try to be nice, even with upset and angry people. Only if they keep bothering tell them to (still in a nice way if possible) go elsewhere
always keep discussions about the project on a mailing list. You don't want to repeat the same discussions over and over again - if you have a mailing list, just point users to the archives before the discussion starts all over again
And you should watch the talk "How Open Source Projects Survive Poisonous People (And You Can Too)" - it's really good and tells you a lot on how to deal not just with 'poisonous people' but also how to deal with all people involved in your project.
I'd like to add that you should make it as easy as possible for your users to get the whole thing running and modify the code - these 'power users' can be 'converted' into developers or at least people who send smaller patches.
Don't try to do it all yourself - for open source projects there are several hosting providers that solve most of the problems. I recommend codeplex or google code.
Setting up build scripts will depend a certain amount on what platform you set up, but in general it's easy to add any tool you want into the script once you start using any sort of build script.
If you really need the one step process you describe, you need a build server. I use TeamCity, which I have set up to watch for any changes in svn and trigger build/test whenever something is checked in. The build server will generally be able to perform any steps that you put into the build script.
Read up on Git as an alternative to SVN
free public repository/bug tracker/wiki/fork-enabled community in Github (which hosts symfony and PHPUnit amongst others)
"How do I handle (technically) submissions from other users? How can I ensure that those submissions must be approved before being integrated?" - with Git, pull what you/your closest team finds most interesting to the master branch
Consistent API
be inspired of other public API:s
only change in major versions
guessable
Interesting for both users & developers
clear goal (your roadmap - excellent)
useful, contra everything else available
easy to use, but still not easy-enough-to-write/maintain-yourself
You could check out either Ant or Phing to build your project. Include CodeSniffer in the build and you'll save time checking for basic formatting errors/differences.
These are all technical tips, about the soft part... treat humans with respect, a lot of interest and be overly excited about their contributions and make them feel that they're not wasting their time. That would appeal to me.
Take a look at Karl Fogel's book on Producing Open Source Software. It probably has everything that you asked.
You should also plan for engaging the community. I'd recommend reading Jono Bacon's The Art of Community [http://www.artofcommunityonline.org/].
You have a great set of ideas to start. You might have to start by trimming them down! Ask yourself what's necessary for a first release.
For automating the builds and tests, the scripting can be done with ant, maven or phing for PHP projects.
You'll probably need a host so you can demo the product. For PHP that is pretty easy to find.
You need an open source hosting provider-- especially github (but also google code, source forge, etc). Github provides bug tracking, default licenses, blog and great mechanisms for accepting changes from the community. Built on git, it facilitates distributed projects quite well.
Although it's good to have a one-step build and install in place, automating integration of others changes probably isn't important (or desirable) off the bat.
Good luck!