How can I improve socket hang up when connecting many devices? - fiware

Question

I am testing to connect many devices to FIWARE in the following environment.
Each component is deployed in a container on a physical server.
+-------------------------------------------------+
|Comet - Cygnus - Orion - IoTAgentJSON - Mosquitto| - device*N
+-------------------------------------------------+
Under the condition that each device transmits data at 1 msg/sec, the following error occurs at IoTAgent when the number of devices is 350.(That is, 350 msg/sec)
{"log":"time=2018-12-16T14:57:24.810Z | lvl=ERROR | corr=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | trans=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=Error found executing update action in Context Broker: Error: socket hang up | comp=IoTAgent\n","stream":"stdout","time":"2018-12-16T14:57:24.81037597Z"}
{"log":"time=2018-12-16T14:57:24.810Z | lvl=ERROR | corr=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | trans=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | op=IoTAgentNGSI.Alarms | srv=n/a | subsrv=n/a | msg=Raising [ORION-ALARM]: {\"code\":\"ECONNRESET\"} | comp=IoTAgent\n","stream":"stdout","time":"2018-12-16T14:57:24.810440213Z"}
{"log":"time=2018-12-16T14:57:24.810Z | lvl=ERROR | corr=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | trans=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | op=IoTAgentJSON.MQTTBinding | srv=n/a | subsrv=n/a | msg=MEASURES-002: Couldn't send the updated values to the Context Broker due to an error: Error: socket hang up | comp=IoTAgent\n","stream":"stdout","time":"2018-12-16T14:57:24.810526916Z"}
The result of the requested ps ax | grep contextBroker command is as follows.
ps ax | grep contextBroker
19766 ? Ssl 29:02 /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -dbhost mongodb-orion-demo -statCounters -statSemWait -statTiming
Question 1: Where is the cause? IoTAgent? Or Orion? Or MongoDB? Or kernel parameter?
Error found executing update action in Context Broker: Error: socket hang up but there is no error log displayed in Orion.
Question 2: How can I improve the processing performance of FIWARE?
Do you need the scale of IoTAgent?
Do you need to consider Orion's parameters?
I need to consider values ​​such as reqPoolSize and maxConnections with reference to the following URL?
https://fiware-orion.readthedocs.io/en/master/admin/perf_tuning/#http-server-tuning
Do you need the scale of Orion?
How to scale Orion GE?
Question 3: Is there a batch operation on IoT Agent?
On the following page, you should do a batch operation instead of opening a new connection for each entity, but is there such a function in IoTAgent?
ECONNRESET when opening a large number of connection in small time period

Answer 1

It is difficult to provide a right answer, as performance depends on many factors specially in complicated setups involving several components interacting among them. However, I'll try to provide some thoughts and insights based in the information you provide and my previous experience.
With regards to Orion, I'd recommend you to have a look to the documentation on performance tunning. Following the indications in that page you can increase the performance of the component.
However, having said that, I don't think that Orion is the cause of the problem in your case, based on:
Even without performance optimization Orion typically reach a throughput in the order of the ~1,000 tps. It should cope updates at 350 tps without problems.
Orion is not showing error logs. The error logs you have are produced by IOTAgent component, as far as I understand.
Thus, focusing in IOTA, maybe it would be better to use IOTA-UL instead of IOTA-JSON. The UL encoding is more efficient that JSON encoding so you can gain in efficiency. In addition, IOTA-UL allows you to send multimeasures (using # as separator) which I don't know if fits your case but can be seen as a limited form of batch update (see UL documentation for more detail).
If that doesn't work another posibility is to send data directly to Orion using its NGSIv2 API. That would have several advantages:
Simplified setup (two pieces less: MQTT broker and IOTAgent)
Under same resource conditions, Orion native performance is typically higher than IOTAgents performance (as I mention before ~1,000 tps or even higher after applying performance optimizations)
NGSIv2 API provides a batch update operation (look for POST /v2/op/update in the NGSIv2 specification document cited above)