Related
I've been considering for some time now to get into be b2b integration business. I've researched the tools available for doing this,
like Oragle's WebLogic Integration, IBM's WebSphere, or Microsoft's BizTalk. They all seem to do the job (each having their ups and downs).
I've also looked at some companies that already are doing this (ex: www.hubspan.com). It seems that b2b integration is very needed service.
Although my background is in integration of commercial products with open source software, I feel that concerning the b2b integration world,
I still feel that I need to feel some blanks.
So basically I'd like to clear a few things concerning all this:
All the frameworks that I previously mentioned are just that, frameworks. They allow to build an application ON TOP of the said frameworks,
they are not itendet ot be a final product. I assume that this is because the integration needs of different companies vary so much,
that an out-of-the-box solution is just not possible. So my question is, do the applications build with the said frameworks vary so much
from business to business, that it's not possible to reuse them?
Also, is it possible to build a single framework of Suppliers and Customers (build a Core of somekind), and connect new Costumers and/or
Suppliers as they come? (this is the way HubSpan did it, not counting the developing of custom Connectors to the Client ERP systems).
Or will I have to do a separate integration for each Customer?
How much work hours is required to complete a typical integration project, (assuming everything is planned and executed properly)?
(For the sake of simplicity, let's say that the integration includes only 'Query Product Price', 'Query Product Availability',
and 'Purchase Order Management'.
And finally, is this a job for a sigle person (can I do this myself?, assuming I have the knowledge to do it) or a team is required?
Thanks in advance for sharing your thoughs and oppinions.
Yes they can vary that much.
It depends on the business. Some will integrate easily while others will need custom modules and connectors
There isn't really a "typical" integration project.
Depends on the size of the project. If you're talking fortune 500 companies then no. If you're talking a local manufacturer and local supply house (presumably small) then maybe.
This is probably a question better asked on the programmers.stackexchange
I think it varies a lot. You should probably define what you mean by B2B, and there are a lot of different types these days. From a BizTalk perspective, it is possible to build an application service provider (ASP) version of B2B but it is hard to do.
The level of customization is one of the factors that drives up cost and the length of the project. I think it is difficult to do B2B alone, usually there is so much business domain knowledge specific to each company that you need those business people to help explain the existing systems.
Thanks,
I really need your help with this. We are planning on developing a real-time web application. We look at different libraries and concepts and a little confused.
What we need is: clients connect to websites and send data(usually an integer + client machine name) whenever they want (usually 1-5 seconds). Also, the same clients must receive data(the data received from other clients) from the server in a real-time mode. (maximum 0.5 seconds). Also, this data must be stored in the database.
We were thinking about using different technologies, but cannot decide which one to use.
We need this web application to be supported on Iphones and Android Phones (maybe blackberry).
and, of course desktop browsers.
Pooling seems not a very good Idea in this situation, due to highloads.
Html 5 web sockets kinda new, and probably not supported by all browsers.
Have anyone used nodejs ?
or twisted matrix: http://twistedmatrix.com/trac/?
or orbited(cannot post more than one link)?
or tornado?
Or XMPP(Jabber. I did not find good examples.)?
or something else?
What technology is the best to use in this type of project? Also, we would probably prefer the technology that has some community support and free to use.
Thanks a lot!
There is a lot of things to consider here. I would say that HTML 5 is not an option, simply due to support across platform.
Running with NodeJS is most likely possible, but the communication methods are really complicated. Pushing data to a page isn't really something that HTML/web apps are designed to do....
To get a valid answer you are going to need to get someone to come in and sit with you to really iron out details and implementation.
When you say that clients "connect to a website", do you really need it to be a website? It sounds like all the client is sending is a number and for that you don't need a website. Just pick the language of your choice, open up a socket, and go from there.
Are you streaming data to be visualized? You might want to take a look at graphite (and/or "pyped" which is part of graphite).
What kind of data? What is the purpose?
For real-time you're not going to get a web site unless you use some type of RIA but even then, it isn't going to be enough. Services aren't going to be good enough either. You're going to end up doing some type of polling which will only ever be psuedo-real-time unless you do duplex mode which wont be supported on most of the platforms you want to support.
sockets are the way to go but that requires a client for each platform you want to handle. Maybe you should rethink your requirements.
I've been interested in 4D SAS' database product for a long time, though have barely touched it in eons.
In considering what tools to use for application development, especially one that will require a database component, what should be looked for when considering open-source tools like MySQL and PostgreSQL vs proprietary solutions like 4D or Pervasive SQL?
What good (and bad!) experiences has the SO community had with various DB tools like 4D, Pervasive, FilemakerPro, etc?
Any bad experiences?
Difficult to make a relevant list of Pros and Cons without a context.
My advice would be the following: when making the decision of using a proprietary database, make sure that this decision is based on strong facts and not merely a technical interest for an exotic tool. Put into the balance the benefits for using the proprietary database and the advantages of a non-proprietary solution.
The answer is different from system to system.
A prerequisite is that your system is well identified, with a clear scope, a quite predictable evolution, so that the results of your analysis will be robust. Then, if your proprietary solution brings a real benefit for your system, that you are comfortable with the support and that you can afford the overall cost, you should be a good candidate for the proprietary solution.
4D is a MacOS/Windows only cross-platform, proprietary database system with both stand-alone and Client-Server varieties. You would do well to compare it to Alphafive.com software which is Windows only. I've worked with it for 17 years and it has served me and my department very well. Off the top of my head ...
Pros:
Interface & code are closely tied to the data engine which makes development of rich, cross-platform user interfaces very fast and easy.
Proprietary relational data engine runs natively on both platforms, along with native client interfaces (but requires licenses for multi-users). Auto-relations are helpful (but sometimes get in the way).
Can access external systems via SOAP and ODBC and SQL drivers (limited).
Can access 4D from external systems via SOAP or http requests & web pages.
Native procedural programming language based on Pascal and is EASY to learn.
Excellent tool for small to mid-sized departments.
Latest version accepts subset of SQL commands AND original data access, so it's backward compatibility record has been very good.
Security is EASY in 4D.
You can build solutions to deploy through a variety of means, and are not limited by whether or not MS Access is installed.
Cons:
Interface & code are closely tied to the data engine which can lead to limited use of abstraction and "black-box" coding unless you make it a goal of your development.
Compiles to one monolithic structure file forcing restart for single fixes.
Language is still only procedural--making it harder for object-oriented programmers to accept. Every method requires separate "file" in 4D so you can't include more then one function or procedure in a single routine -- it will take some getting used to it.
While company appears to be in good shape, growing and developing, you simply never know as they keep their condition to themselves.
Company has never really marketed itself--trusting in its developer base to spread the word and grow the product through site deployments and product upgrades. Web site is clearly useful only to developers who already use the product -- it simply fails to attract new users.
Product upgrades have always seemed to focus on how the tool is better for the DEVELOPERS rather than for the CUSTOMERS of those developers.
SQL lacks views, compound indexes, and other common SQL features.
When a user requests a report of specific columns of data, I often have to write yet another program just to provide that specific data -- I can't always just query the data and generate a text file.
Does not handle new OS versions with nearly the ease of web browser based applications. Older version is broken on Mac OS 10.6, and newest version requires the latest Mac OS 10.6. No version is certified yet on Windows 7.
I've been nearly a year at learning ASP.NET and a few weeks at Ruby on Rails. While SQL data stores are EASY, user interface is HARD -- but worth it when your application still functions through OS upgrades. You can always use an older browser if the latest version breaks something.
I'd recommend you consider either of those, depending on how much funds you have available to implement the project--Rails being the cheaper of the two. Then, ANY system with a web browser can access the data, and you can fix interface pages on the fly as needed rather than taking the whole system down a few minutes for a single, simple update. Those skills might be more marketable in the future.
I will only say one thing.. Watch the "actual" cost of your decision.. Most proprietary database systems are Windows only.. or sometimes Mac/Windows only.
This means that along with paying quite a bit of money for the database system, you must also pay a good amount of money on a Server operating system to run it...
Also, compare the database system with current open source solutions. Is it really worth it? After moving from Microsoft Sql Server(which has a free edition, but anyway) to PostgreSQL I was blew away that people pay so much for SQL Server.. I mean, Postgres to me is a lot more clean, and most of it works exactly how you'd expect(unlike in certain SQL server syntaxes) and it has more features built into it(programming stored procs in Ruby anyone?)
So basically, compared the proprietary with the open source software and decide upon which one to take by total pricing(including OS) and feature set..
Pro of zeroing in on any DB: it's got good non-portable features that help you get things done
Con of zeroing in on any DB: sometimes a different DB is appropriate (for example running your tests with in-memory SQLite instances), but that option is now closed
Con of a proprietary commercial DB: if you need many instances, licensing costs can kill you
Consider the following questions:
How easy (or difficult) is it to make changes in maintenance? Applications are likely to spend far more time in maintenance than they do in development, so if changes are hard, long-term pain is guaranteed.
What is the quality of support? A system that is well-documented, proprietary or otherwise, is going to be easier to work with.
How large (or small) is the user community? Systems with larger user communities mean more people to ask for assistance if and when things go wrong.
How robust are the import/export capabilities of this proprietary database system?
I found the last point particularly useful at my first full-time job. Our client was using CA-Ingres, and no one at the company knew it well enough to write queries to validate the data. So I came up with the idea of exporting the data from Ingres and importing it into MS SQL Server (which I knew from a brief stint at Sybase Professional Services) so we could write our validation queries there. If it had been really hard to export data from Ingres, my idea wouldn't have been an option at all.
From 4D's webpage, I gather that we are looking at a complete development+deployment environment, not a standalone database as such. So the alternatives you could be looking at include stuff like django, ruby-on-rails, hibernate and others. The real question, of course, is if the proprietary system can save you enough money doing the product lifetime to justify the costs of the product. And that would depend on the type of human resources you have available.
4D is a good option for vertial applications. I have worked for a company which used 4D to build a medical records and billing application for general practitioners and specialists. The rapid design and deployment features of 4D enabled the application to quickly move with market desires and legislated changes to medical record storage.The environment itself was not cutting edge, but it was integrated, cross platform and very productive.
If you are entering a market with high vendor lock-in and a high barrier to entry, then I think proprietory integrated development environments are a good option.
At various points in my career, I've used and gotten very good at FileMaker Pro, FoxPro, 4D, and a few other commercial products. Now I mainly use PHP/MySQL, and haven't used the latest versions of any of the products.
I've always liked FileMaker because most people who can use a computer can pick up FileMaker and design their own systems. They don't have to know programming or database design. But, you can "program" FileMaker, put a web front end on it, or do other more sophisticated setups if you need to. Many times I was "handed" a system created in FileMaker by a non-technical person that needed to be made into a full fledged data management system. The good part was that all the "specs" and data flow were already designed into a system. The prototype was already created!
4D and FoxPro I always found required a certain amount of extra programming and/or database knowledge to really do anything with. 4D & FileMaker are really complete self-contained systems, not just database systems. Although they all have the ability to hook into other backend databases systems (i.e. MySQL, Oracle), that is not their strong point.
On the downside, doing more complex, dynamic systems can be difficult in 4D and Filemaker due to everything being tightly coupled. Because of their cost, you really would want to create multiple systems with them. Which means you need to really "buy into them" to get your money's worth.
The key concept is always adherence to standards: if you plan to use 4D's custom and / or special designed functions (but the discussion could be far more general, and cover any other free or commercial tool in the wild), well, just use it and take your advantage.
Not surprisingly, that's why huge DB systems like Oracle or IBM's DB2 in the past were wide accepted for specific business areas, as commercial transactions, for instance.
The other main reason to adopt a very closed solution is the legacy support. One of the products you cited (Pervasive SQL) acted as a no-effort port for BTrieve-based applications in late 90s, and it gained popularity thanks to the huge BTrieve community all over the planet.
Finally, last but not least, you should evaluate the TCO (Total Cost of Ownership) not only in terms of license price (single seat, network environment, site licenses and so on), but also for what concerns tech support, updates and availability for your platform. Many business units I know have been obliged to change their base OS for DB related problems.
Tip: add a bonus for custom solution that are proven or supported for usage in virtualized environments, if you aren't in seek for extreme performances. It will save more than a head ache for your DB manager.
In all other cases, rely on opensource/freesoftware DBs. MySql and Postgres for big projects, SQLite for single app persistence layer. Fairly standard and very good (community) support. Good value for no price.
I don't have any experiences with the proprietary database products you listed: 4D, Pervasive, FilemakerPro.
I'd be interested in knowing what those products offer that make them more attractive to you than the open source alternatives, you listed: MySQL and PostgreSQL.
I'd be interested in what makes those more attractive to you than the much more popular proprietary alternatives: Oracle, SQL Server, DB2, etc.
Without you providing more specifics, it's hard to advise you.
I personally feel safer using a widely used open source solution than a narrowly used closed source solution. The more widely used, the more battle-tested it's likely to be. The more open, the more control over my own destiny I have in case I do encounter some bug.
I have reported bugs to open source projects and gotten a quick fix. I have reported bugs to companies that make for-profit proprietary software and have gotten nothing.
I'm currently writing a large scale ASP.Net web app.
One of the thngs I can't find out about is how to justify when to use the cloud. E.g. when should I use google app engine/azure?
Also, when would I want to use bigtable over a standard dbms such as Sql Server?
Thanks
Cloud computing is all about scalability. It allows you to scale up AND scale down without having to rework your designs.
It works well for small sites, since you are only paying for resources used, but if you need to scale up, it just happens automatically (provided your application was designed for the cloud).
Also, there are theoretically much better tools in place for maintaining uptime and reliability in the cloud. For example, a system upgrade can happen without stopping your service, since the cloud computing platforms can automatically take on or off servers to service your application.
There's been a lot of talk about that from the Azure devs.
Also, there can be a financial motivation for using the cloud. Using a hosted cloud architecture can be less expensive than managing the multiple servers (DB, web, etc) that would be required for a traditional site, at least up front. As your usage goes up, the cost follows, but in theory, it can be more cost effective.
I'm not too familiar with anything else except app engine and EC2.
I'll try to add something to the previous answers:
The best thing about app engine is it's free until you attract a certain amount of users and you are charged for what your application uses, idle time is not charged.
Big table may differ from an rdbms architecturaly but from a perspective of a developer using it it's not that different.
Another good thing is python is supported. The bad thing is the standard library is crippled.
Also, you don't have full control over your data on the cloud (appengine), what I mean is you can't completely restrict the people from google from taking a peek in what you store there.
This question is very closely related to another question asked today:
"When shouldnt-you-use-a-relational-database?"
Relational databases and non-relational databases (like BigTable) address different needs. Not only in scale and performance, but in the structure and usage of the data.
The "Cloud" as I understand it is about scalability primarily. That is, the architecture refers to a capability to increase capacity in a scalable way.
Also, the Cloud is used frequently in reference to the Software-as-a-Service (SaaS) model, where someone else takes care of the servers, but that's an independent issue from the Cloud architecture. I.e. you could operate your own set of servers in a Cloud architecture.
So the justification for using the Cloud architecture is that you have an application that has a variable need for computing capacity. So it would be overkill to have N servers dedicated to match your peak level of activity. The Cloud allows you to vary your usage of the servers as your level of activity grows (and diminishes) over time.
The justification for using a SaaS model is that you don't want to be in the business of operating a data center. You're willing to relinquish some control and pay for the service, so that you can leave operation details to the experts in that technology. They handle backups, hardware failures, upgrades, 24x7 operation, etc. You handle your application and your business.
I recommend you subscribe to and read the High Scalability blog, especially some of the most visited posts such as those about the architecture of various large sites, as you will learn a lot from it that may help you make a decision. There is no hard rule as to when you should or should not use a cloud service or move from a relational database to a keyvalue system like BigTable.
One upside of cloud services in any case is that if you build your application with them, it will be immediately scalable and require much less rework later on if you require that kind of performance. However, in view of premature optimisation, it would be wise to be sure that you need that kind of scalability before you decide to build your app on such a platform.
There are several concepts to wrap your head around when using a datastore system like BigTable as well, such as not being able to just slam out writes like you would in a relational database, and having to precalculate a lot of your data rather than just doing that based on info from the database.
Although again, you can learn a lot from reading the abovementioned blog and related posts about Youtube, Plentyoffish, Google, etc.
You say you are "currently writing a large scale ASP.NET app". If you have made significant progress on it, you are already pass the point where you can justify using Google app engine or Azure. Both require significantly different architectures than you have build with a traditional application due to language support, database differences, and maturity.
Google App Engine is Python only so switching to it would require a complete rewrite
Big table is not a relational database and requires very different coding patters. SQL Data Services originally announced to be non-relational as well, but is moving to be more relational. I have not seen how close to a standard MSSQL database it currently is.
I would consider Google app engine to be a relatively immature platform so far. Database functionality is limited, you cannot run background processes, profiling and performance tuning tools are limited at best. Azure is currently in limited community preview, and so is not even available to ship a product on today.
While there are many very valid reasons to use a cloud architecture, moving to it will require significantly different architectures. Think about what effect changing that architecture (and possibly waiting for platform availability) will do to your release date.
If you are early in your project, cloud vs. not cloud is a great question to ask. If you have well on your way, I think that the importance of getting to shipping code and leveraging the work you have already put in should trump any benefits to the cloud you may see.
I friend asked me this today.
Picasa Web has a cool (and frightening :-) feature where it will recognize all the faces in your photo album.
But the PC (desktop) version doesn't have this.
Several reasons I can think of:
They just haven't gotten around to writing the PC version of the code.
They are licensing that feature and it costs a lot more (or isn't available) on the PC.
Takes a lot of processing power (this seems odd b/c MY PC cycles are free to Google, but they have to pay for for cycles consumed on their server.
Any other thoughts?
I'm certain it'll make it out in coming releases but Google is a funny company when it comes to its own competing/complementing services. One thing is for sure, only somebody on the Picasa team could give an accurate answer.
But we could hypothesise several things...
They don't want their code reverse-engineered.
(As you say), they aren't licensed to redist
It's blocked in the dev version by other new features that aren't complete yet
They don't want to release it because they want people to use PicasaWeb as a social photo network.
I don't think processing power is an issue. If they're running it in bulk on their own servers for free, a modern desktop could probably run it without issue.
From my limited contact with face recognition software, it's probably the redistribution issue. When I dealt with it, face recognition was its own little world with extremely high per-CPU licensing costs and tremendous paranoia about code getting loose.
I'm not so sure it's not a processing issue. It took Google's massive servers 30 minutes to run through all my photos. I can only imagine that same task would have taken days on my local machine.
Actually, its in, just in limited functionality when you do a search, there's an icon to find only photos with faces. The experimental passport feature also works that way.
So the answer is:
Not the same base (APIs) available or used and not the same language so its not directly portable.
Not the same software and there are no stated goals to make both apps feature equivalent.
Programmers are limited and their time is too. They make choices as to what implement now.
No idea if this is the case for Picasa, but there's another case where licensing could be the issue. If the server-side code is using code with a restrictive license with DRM (GPL, for example) which restricts how you can distribute modules using the code. Running that module on a web server, where the user only gets the output, is legal under such licenses. If that code was distributed, there would be many legal requirements attached which would likely be very undesirable for commercial software companies, including google. This is one very good reason to have some capabilities only accessible through web services.
This was also the case with Riya (who was arguably the first to market with reliable facial recognition for consumer photo collections).
The biggest reasons are likely:
Processing Time (they can't control
how fast your CPU is and therefore
they can't control the experience).
Facial recognition is very likely to
be process intensive (this was Riya's
stated reason for not doing it
client-side)
The recognition process requires a
LARGE volume of data for processing
that is only accessible on the
server? (In other words, the process needs to spin through millions of faces, not just the faces that you have on your hard drive?)