Problem Set
Assuming an application with two networking classes (these are the backend for every outgoing and incoming network connection of the application), I would like to monitor the network load created by the application itself in Python 2.7.
Already did multiple searches on SO and the Net but did not get the results or ideas i was looking for.
I would like to achieve the solution without 3rd-party application dependencies (such as Wireshark or similar products as I am not in control of end-user OS. Application shall be 100% Cross Platform).
networking classes:
mysql-Driver based on mysql-connector-python.
"general" networking class to check availability of hosts using socket library shipped with python
Question(s)
Are there python libraries that can achieve this without any 3rd party product beeing installed?
is the approach so off that another approach should be much easier / more likely possible?
Related
I am trying to integrate our third party application with NetSuite. I want to be able to import sales invoice details generated from our third party system (which uses REST API) into the NetSuite invoice form.
The frequency of import is not too crucial- an immediate import will be ideal, but sending data once a day is fine as well.
I want to know what I have to use to do this API integration - SuiteTalk, RESTlet or Suitelet.
I am completely new to this topic and after a few days of research, I learned that there are 3 options for an API integration with netsuite (Suitelets, restlets and suitetalk which comprises REST and SOAP based web services). I also learned that there are scheduled scripts and user events, but I'm not too clear on the idea.
I need some help identifying which integration option I should choose.
Any and all information about netsuite API integration is appreciated!
I would avoid REST/SOAP. SOAP is outdated, and REST is incomplete and difficult to use.
Suitelet's are for when you want to present your own custom UI to frontend users, like a special new kind of custom form not relevant to any particular record. Probably not what you want.
What you probably want is to design a restlet. A restlet is a way for you to setup your own custom url inside NetSuite that your program can talk to from outside NetSuite. Like a webpage. You can pass in data to the restlet either inside the URL, or inside the body of an HTTP request (e.g. like a JSON object), and you can get data back out from the body of the HTTP response.
A restlet is a part of SuiteTalk. The method of authenticating a restlet is the same for the method of authenticating a request to the REST API. So, learning about SuiteTalk is helpful. The code you use to write the restlet, SuiteScript, is the same kind of code used to write suitelets and other kinds of scripts.
So you will want to learn about SuiteTalk, and then, in particular, SuiteTalk restlets.
this is a really subjective issue.
It used to be that SOAP/SuiteTalk was a little easier in terms of infrastructure and since Netsuite's offerings are ever changing the REST/SuiteTalk might fill this space in the future.
Since Netsuite deprecated the Full Access role setting up integrations almost always involves the integrator having to provide a permissions spec. The easiest way to do that is via a Bundle. For token based authentication (TBA) there also needs to be an integration record from which you need Consumer Id and Secret Tokens.
So as of this writing the set up for SOAP/SuiteTalk and RESTLets is roughly the same. The easiest way to communicate these is with a bundle so if you are a Netsuite dev with a dev account you can set these up in a bundle and have your customer import them.
So equal so far but differences:
SOAP/Suitetalk is slow. IMO not suiteable for an interactive interface
SOAP/Suitetalk the code is all in your external app so changes to the code don't require any changes in the target account.
RESTlets can be pretty speedy. I've used these for client interactions.
Updates require re-loading your bundle or overwriting your bundle files in the target account (with the resulting havoc if an admin refreshes the bundle)
RESTlets give you access to the features of the account on which you are running so that code can run appropriate chunks For instance features such as matrix items, multi-location inventory, one-world, pick/pack/ship, volume pricing, multi-currency will all change the data model of the account your code is running against. RESTlets can detect which features are enabled; SOAP/SuiteTalk cannot.
So really the only advantage at this point that I see for SOAP/Suitetalk is that code updates don't require access to the target account.
Who is making the changes? If it is your NetSuite developers, then your options are SUITELET or RESTLET.
If its your third-party application team, they own the code and the process and do all their work sitting outside of NetSuite - your option is SUITETALK/SOAP. Of course, they need to know something about NetSuite, but your business analyst would be sufficient to support them. As of 2020.1+, there is also support for native REST APIs in addition to SOAP in case you still want to use REST, but not write your own RESTLETS.
As the above comments mention, Suitetalk does perform a little slower than calling RESTLETS. So that maybe one of the deciding factors.
You may consider SUITELETs for integration only if you want to bypass all authentication schemes, by setting the suitelet as public. Highly inadvisable though.
If the third-party application supports REST APIs, you could call them directly from within NetSuite - either from user events or from scheduled scripts.
You can also consider iPAAS platforms like Dell Boomi, Celigo, Jitterbit, etc. These are general-purpose integration platforms, and make connecting one platform to another easy, with minimal coding. If your Company is already invested in these iPAAS platforms for other enterprise applications, then the choice is that much simpler.
I've been looking into sockjs-tornado recently and am working on a chat function for a social networking site. I'm trying to get a feel for common methods used in building scalable multiroom chat functionality. I'll outline a couple of the methods I've thought of and I'd like to get feedback. What methods are used in the real world? What are the advantages and disadvantages to these methods?
Prereqs:
running tornado
using sockjs-tornado lib
sockjs-client lib for js
Everything else is open.
Methods I've considered:
For loop
This seems like the simplest way to go. You create a user class that subscribes to certain room classes. The user sends a message class that contains a room id and the server redirects the message in the loop only to users that have subscribed to that room. This seems to me to be by far the worst because the complexity is obviously at least linear. (Imagine 500 users connected at once to 5 chat rooms each.)
Multi-tasking/multiple server instances
This also seems like a bad idea because you could have 500 server instances running at any time on... different ports? I'm really not sure on the implementation of this method.
Native support
Now granted, a lot of libraries have this built in such as socketio. However that's not an option due to the sole node.js support. (I'm on tornado server.) Socks in particular does not have built in support for multiple "rooms".
Conclusion
I'm looking for resources/case studies, and industry standards. Any help would be appreciated.
I would just use a message queue server like RabbitMQ with a fanout exchange as each "chat room".
You can see an example of using a fanout exchange in Python here.
The Pika AMQP library works with Tornado, too.
The advantage with using a message queueing system is that you can have users connected to different Tornado processes on different servers while still being in the same "room", giving you high availability on the HTTP layer.
RabbitMQ also has HA capabilities (although not the greatest).
We have about about six systems (they are all internal systems) that we need to send data between. Currently we do not have a consistent way of doing this. We use SSIS, SQL Server linked servers to directly update databases, ODBC connections to directly update databases, text files, etc..
Our goals are:
1) Have a consistent way of connecting applications.
2) Have a central way of monitoring and logging the connections between
applications.
3) For the applications that offer web services we
would like to start using them instead of connectiong directly with
the database.
Whatever we use will need to be able to connect to web services, databases, flat files, and should also be able to accept data via a tcp connection.
Is Biztalk a good solution for this, or is it is overkill?
It really depends. For the architecture you're describing, it would seem a good fit. However, you will need to validate wether biztalk can communicate whith the systems you are trying to integrate. For example; when these systems use webservices, message queues or file based communication, that may be a good fit.
When you start with biztalk, you have to be willing to invest in hardware, software, en most of all in learning to use it.
regarding your points:
1) yes, if you make sure to encapsulate the system connectors correctly
2) yes, biztalk supports this with BAM
3) yes, that would match perfectly
From what you've described (6 systems), it is definitely a good time to investigate a more formalized approach to integration, as you've no doubt found that in a point to point / direct integration approach will result in a large number of permutations / spaghetti as each new system is added.
BizTalk supports both hub and spoke, and bus type topologies (with the ESB toolkit), either of which will reduce the number of interconnects between your systems.
To add to oɔɯǝɹ:
Yes - ultimately BizTalk converts everything to XML internally and you will use either visual maps or xslt to transform between message types.
Yes. Out of the box there are a lot of WMI and Perfmon counters you can use, plus BizTalk has a SCOM management pack to monitor BizTalk's Health. For you apps, BAM (either TPE for simple monitoring, but more advanced stuff can be done with the BAM API).
Yes - BizTalk supports all the common WCF binding types, and basic SOAP web services. BizTalk's messagebox can be used as a pub / sub engine which can allow you to 'hook' other processes into messages at a later stage.
Some caveats:
. BizTalk should be used for messages (e.g. Electronic Documents across the organisation), but not for bulk data synchronisation. SSIS is a better bet for really large data transfers / data migration / data synchronisation patterns.
. As David points out, there is a steep learning curve to BizTalk and the tool itself isn't free (requiring SQL and BizTalk licenses, and usually you will want to use a monitoring tool like SCOM as well.). To fast track this, you would need to send devs on BizTalk training, or bring in a BizTalk consultant.
. Microsoft seem to be focusing on Azure Service Bus, and there is speculation that BizTalk is going merged into Azure Service Bus at some point in future. If your enterprise strategy isn't entirely Microsoft, you might also want to consider products like NServiceBus and FUSE for an ESB.
You problem is a typical enterprise problem. Companies start of building isolated applications like HR, Web, Supply Chain, Inventory, Client management etc over number of years and once they reach a point these application cannot be living alone and they need to talk to each other, typically they start some hacked solution like data migration at database level.
But very soon they realize the problems like no clear visibility, poor management, no standards etc and they create a real spaghetti. The biggest threat is applications will become dependant on one another and you lose your agility to change anything. Any change to system will require heavy testing and long release cycle.
This is the kind of problem a middleware platform like BizTalk Server will solve for you. Lot of replies in the thread focused on cost of BizTalk server (some of the cost mentioned are not correct BTW). It's not a cheap product, but if you look at the role it play in your organisation as a central middleware platform connecting all the applications together and number of non-functional benefits you get out of the box like adapters to most of the third party products like SAP, Oracle, FTP, FILE, Web Services, etc, ability to scale your platform easily, performance, long running work flows, durability, compensation logic for long running workflows, throttling your environment etc., soon the cost factor will diminish.
My recommendation will be take a look at BizTalk, if you are new then engage with local Microsoft office. Either they can help or recommend a parter who can come and analyse your situation.
As our systems grow, there are more and more servers and services (different types and multiple instances of the same type that require minor config changes). We are looking for a "cetralized configuration" solution, preferably existing and nothing we need to develop from scrtach.
The idea is something like, service goes up, it knows a single piece of data (its type+location+version+serviceID or something like that) and contacts some central service that will give it its proper config (file, object or whatever).
If the service that goes online can't find the config service it will either use a cached config or refuse to initialize (behavior should probably be specified in the startup parameters it's getting from whom or whatever is bringing it online)
The config service should be highly avaiable i.e. a cluster of servers (ZooKeeper keeps sounding like a perfect candidate)
The service should preferably support the concept of inheritence, allowing a global configuration file for the type of service and then specific overrides or extensions for each instance of the service by its ID. Also, it should support something like config versioning, allowing to keep different configurations of the same service type for different versions since we want to rely more on more on side by side rollout of services.
The other side of the equation is that there is a config admin tool that connects to the same centralized config service, and can review and update all the configurations based on the requirements above.
I know that if I modify the core requirement from serivce pulling config data to having the data pushed to it I can use something like puppet or chef to manage everything. I have to be honest, I have little experience with these two systems (our IT team has more), but from my investigations I can say it seemed they are NOT the right tools for this job.
Are there any systems similar to the one I describe above that anyone has integrated with?
I've only had experience with home grown solutions so my answer may not solve your issue but may help someone else. We've utilized web servers and SVN robots quite successfully for configuration management. This solution would not mean that you would have to "develop from scratch" but is not a turn-key solution either.
We had multiple web-servers each refreshing its configurations from a SVN repository at a synchronized minute basis. The clients would make requests of the servers with the /type=...&location=...&version=... type of HTTP arguments. Those values could then be used in the views when necessary to customize the configurations. We did this both with Spring XML files that were being reloaded live and standard field=value property files.
Our system was pull only although we could trigger a pull via JMX If necessary.
Hope this helps somewhat.
Config4* (of which I am the maintainer) can provide you with most of the capabilities you are looking for out-of-the-box, and I suspect you could easily build the remaining capabilities on top of it.
Read Chapters 2 and 3 of the "Getting Started" manual to get a feel for Config4*'s capabilities (don't worry, they are very short chapters). Doing that should help you decide how well Config4* meets your needs.
You can find links to PDF and HTML versions of the manuals near the end of the main page of the Config4* website.
Problem:
I need to design a networking application that can handle network failures and switch to another network in case of failure.
Suppose I am using three ethernet connections and one wireless also . At a particular moment only one connection is being used.
How should I design my system so that it can switch to another network in case of failure.
I know this is very broad question but any pointers will help!
I'd typically make sure that there's routing on the network and run one (or more) routing protocol instances on the host. That way network failure is (mostly) transparent to the application, as the host OS takes care of sending packets the right way.
On the open-source side, I have good experiences with zebra and quagga, at least on linux machines.
Create a domain model for this, describing the network elements, the kind of failures you want to be able to detect and handle, and demonstrate that it works. Then plug in the network code.
have one class polling for the connection. If poll timeout fires switch the ethernet settings. For wireless, set the wifi settings to autoconnect and then just enable/disable the wificard.
(But I dont know how you switch the ethernet connection)
First thing I would do is look for APIs that will give me network disconnection events.
I'd also find a way to check the state of the network connections.
These would vary depending on the OS and the Language used so you might want to have this abstracted in your application.
Example:
RegisterDisconnectionEvent(DisconnectionHandler);
function DisconnectionHandler()
{
FindActiveNetworkConnection();
// do something else...
}
A primitive way to do it would be to look out for network disconnection events. Your sequence would be:
Register/poll for network connections status changes. Maintain a list of all active network connections.
Use the first available network connection (Alternately you could sort it based on interface bandwidths, and use the one with highest bandwidth).
When you detect a down connection, use the next active one.
However, if there are implications to the functionality of your application, based on which network connection you use, you are much better off, having either a routing protocol do the job for you, or have a tracking application within your application. This tracking application would track network paths (through various methods like ping, traceroute, etc) across all your available interfaces to see which one can reach the ultimate destination, and use the appropriate network interface.
Also, you could monitor your network interfaces for not just status changes, but also for input/output errors, and change your selection accordingly. This would help you use the most efficient network at any given point of time. But this would need to be balanced with the churn caused by switching a network connection.
If you control all of the involved hosts, Multipath TCP will probe all of your connections and automatically choose the one that works; if multiple connections are working, it will load balance across them.
If you don't control the endpoints, there's no choice but doing the probing in the application. Mosh is an example of an application that does that quite elegantly.
You didn't mention what your application does; perhaps it would be possible to redesign your protocol so that it uses all available connections simultaneously, the way BitTorrent does, and therefore doesn't care about some links being down at any given time?