I used to install, configure and use Airflow daily. I have a new project and it's going to be the first time I install airflow 2.0 from scratch.
Reading the documentation about the difference between the two versions I saw that in the extra packet we can install with airflow "async" is available (it's not new and it's available for the v1). I never face a tutorial or a medium article talking about using "Async worker classes for Gunicorn" like you can read in the documentation.
Do anybody can explain to me the benefit for this extra package ?
The Gunicorn HTTP Server has different classes of workers: sync, async, etc. Based on the expected load, it may be beneficial to chose one worker class over the other. For example, the assumption behind choosing the sync class is that the web application (requests to which will be served by the sync workers) is resource-bound in terms of CPU and network bandwidth.
Airflow webserver uses the sync class by default.. I don't have a real-world example in favor of the async class. But I could imagine some custom plugin that extends the Airflow's webserver and has some long-lasting IO-bound logic might benefit from async workers. Although, this is just a guess...
Related
Is it good idea to show/hide React component using window.env
for example we have feature which we are not ready to release yet,so we are thinking of hiding it using window.env.FEATURE_ENABLED=0 (these vars will be picked by api call to service that serves bundle to browser)
But,I am thinking its risky since user can look at windows.env and set window.env.FEATURE_ENABLED=1 and start seeing the workflow which we intend to hide.
Could anyone please provide their take on this.
Yes, it could potentially be risky for the reason you say.
A better approach would be to only include finished features in the production build - unfinished features that are still in testing should not be sent to the client. For such features, have a separate build. Host it:
On a local dev server (usually one running on the developer's personal machine) (great when one is making rapid changes), or
On a staging server - one that's accessible to all developers, and works similarly to the live site, but isn't the same as the production URL
A staging server is the professional approach when multiple devs need access to it at once. It can take some work at first to integrate it into your build process, but it's worth it for larger projects.
I am trying to use the Hyperledger Caliper framework to measure the performance of a blockchain network based on Hyperledger Fabric, containing 3 nodes and 1 orderer and solo consensus.
I already have the network installed and functional with smart contracts responding correctly on different remote virtual machines. I know I have to create a network configuration file and one for testing configuration. At this point my doubts begin. In all the examples I saw, in this configuration file, javascripts files for testing are related, but my smart contract was written in golang.
Must my tests be written using javascript? Can I reference a golang file in this file? Would anyone have an example to provide me? I've been researching for weeks but I can't understand the examples provided by the framework.
Could anyone give me any help, even if it is a link that I haven't seen yet to search further.
The key to caliper's javascript is to call the peer. In other words, it is not executed by directly connecting a specific smart contract (golang).
Caliper(javascript) <-> Peer <-> Chaincode(golang)
caliper requests the peer's chaincode (javascript) to peer, and the peer receives the request, executes chaincode (golang), and returns the result.
That mean, it is completely independent of the chaincode language.
See the link below.
In the case of the code that calls the chaincode in the actual caliper's JavaScript, only the chaincode name, function, and input parameters are required.
hyperledger-caliper-fabric-tutorial
(ex)
...
const contractID = fabcar;
const chaincodeFunction = 'createCar';
const invokerIdentity = 'Admin#org1.example.com';
const chaincodeArguments = [assetID,'blue','ford','focus','jim'];
const result = await bc.bcObj.invokeSmartContract(ctx, contractID, version, myArgs);
console.log(result);
All caliper workloads currently execute in node.js which is predominantly a javascript engine (Chaincode can be written in any fabric supported language). Support for other languages may come in the future either natively or maybe through alternative just a transpiling to javascript or compiling to wasm as the node.js engine increases it's wasm capabilities.
I would recommend looking at the latest tutorial for fabric which can be found here https://hyperledger.github.io/caliper/v0.4.2/fabric-tutorial/
As there are some big improvements to caliper (over the v0.3.2 version)
I read "Those who would like to enjoy the binding, presentation model structuring, testing capabilities, toolkit independence, and all the other benefits of OpenDolphin, but prefer REST (or other) remoting for data access, can use OpenDolphin with the in-memory configuration"
But I could not find any further hints in the docs?
I can't rely on sticky sessions in my load balanced webserver.
Therefore I need to plugin something different for the http session state.
Is there a opendolphin config property prepared for this? If not are there any plugin points available?
since OpenDolphin and Dolphin Platform use the remote presentation model pattern to synchronize presentation models between client and server you need a state on the server. Currently this state is defined in the session. As you said it's no problem to use load balancing with sticky sessions to provide several server instances. If you need dynamic updates between the clients a distributed event bus like hazelcast will help.
Therefore I need to plugin something different for the http session
state.
What do you need? With the last version (0.8.6) of Dolphin Platform you can access the http client in the client API and provide custom headers or cookies. Will this help? Can you please tell us what you need or open an issue at the Dolphin Platform github repo?
I've been looking into sockjs-tornado recently and am working on a chat function for a social networking site. I'm trying to get a feel for common methods used in building scalable multiroom chat functionality. I'll outline a couple of the methods I've thought of and I'd like to get feedback. What methods are used in the real world? What are the advantages and disadvantages to these methods?
Prereqs:
running tornado
using sockjs-tornado lib
sockjs-client lib for js
Everything else is open.
Methods I've considered:
For loop
This seems like the simplest way to go. You create a user class that subscribes to certain room classes. The user sends a message class that contains a room id and the server redirects the message in the loop only to users that have subscribed to that room. This seems to me to be by far the worst because the complexity is obviously at least linear. (Imagine 500 users connected at once to 5 chat rooms each.)
Multi-tasking/multiple server instances
This also seems like a bad idea because you could have 500 server instances running at any time on... different ports? I'm really not sure on the implementation of this method.
Native support
Now granted, a lot of libraries have this built in such as socketio. However that's not an option due to the sole node.js support. (I'm on tornado server.) Socks in particular does not have built in support for multiple "rooms".
Conclusion
I'm looking for resources/case studies, and industry standards. Any help would be appreciated.
I would just use a message queue server like RabbitMQ with a fanout exchange as each "chat room".
You can see an example of using a fanout exchange in Python here.
The Pika AMQP library works with Tornado, too.
The advantage with using a message queueing system is that you can have users connected to different Tornado processes on different servers while still being in the same "room", giving you high availability on the HTTP layer.
RabbitMQ also has HA capabilities (although not the greatest).
As our systems grow, there are more and more servers and services (different types and multiple instances of the same type that require minor config changes). We are looking for a "cetralized configuration" solution, preferably existing and nothing we need to develop from scrtach.
The idea is something like, service goes up, it knows a single piece of data (its type+location+version+serviceID or something like that) and contacts some central service that will give it its proper config (file, object or whatever).
If the service that goes online can't find the config service it will either use a cached config or refuse to initialize (behavior should probably be specified in the startup parameters it's getting from whom or whatever is bringing it online)
The config service should be highly avaiable i.e. a cluster of servers (ZooKeeper keeps sounding like a perfect candidate)
The service should preferably support the concept of inheritence, allowing a global configuration file for the type of service and then specific overrides or extensions for each instance of the service by its ID. Also, it should support something like config versioning, allowing to keep different configurations of the same service type for different versions since we want to rely more on more on side by side rollout of services.
The other side of the equation is that there is a config admin tool that connects to the same centralized config service, and can review and update all the configurations based on the requirements above.
I know that if I modify the core requirement from serivce pulling config data to having the data pushed to it I can use something like puppet or chef to manage everything. I have to be honest, I have little experience with these two systems (our IT team has more), but from my investigations I can say it seemed they are NOT the right tools for this job.
Are there any systems similar to the one I describe above that anyone has integrated with?
I've only had experience with home grown solutions so my answer may not solve your issue but may help someone else. We've utilized web servers and SVN robots quite successfully for configuration management. This solution would not mean that you would have to "develop from scratch" but is not a turn-key solution either.
We had multiple web-servers each refreshing its configurations from a SVN repository at a synchronized minute basis. The clients would make requests of the servers with the /type=...&location=...&version=... type of HTTP arguments. Those values could then be used in the views when necessary to customize the configurations. We did this both with Spring XML files that were being reloaded live and standard field=value property files.
Our system was pull only although we could trigger a pull via JMX If necessary.
Hope this helps somewhat.
Config4* (of which I am the maintainer) can provide you with most of the capabilities you are looking for out-of-the-box, and I suspect you could easily build the remaining capabilities on top of it.
Read Chapters 2 and 3 of the "Getting Started" manual to get a feel for Config4*'s capabilities (don't worry, they are very short chapters). Doing that should help you decide how well Config4* meets your needs.
You can find links to PDF and HTML versions of the manuals near the end of the main page of the Config4* website.