Fiware-quantumleap has issues getting and changing the updated values - fiware

The values sent to the Fiware IoT-agent-json are being successfully updated in CB. But, quantumleap 0.7.6 won't get them anymore (It was working just fine and cratedb was storing the recieved data normally)..the sanity check shows that It's healthy..I think something is going wrong with the DB. The logs shows something like this..
ERROR:app:Exception on /v2/notify [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.8/site-packages/connexion/decorators/decorator.py", line 48, in wrapper
response = function(request)
File "/usr/local/lib/python3.8/site-packages/connexion/decorators/uri_parsing.py", line 144, in wrapper
response = function(request)
File "/usr/local/lib/python3.8/site-packages/connexion/decorators/validation.py", line 184, in wrapper
response = function(request)
File "/usr/local/lib/python3.8/site-packages/connexion/decorators/validation.py", line 384, in wrapper
return function(request)
File "/usr/local/lib/python3.8/site-packages/connexion/decorators/parameter.py", line 121, in wrapper
return function(**kwargs)
File "/src/ngsi-timeseries-api/src/reporter/reporter.py", line 189, in notify
trans.insert(payload, fiware_s, fiware_sp)
File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 179, in insert
res = self._insert_entities_of_type(et,
File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 273, in _insert_entities_of_type
self._update_metadata_table(table_name, original_attrs)
File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 414, in _update_metadata_table
self._store_medatata(table_name, persisted_metadata)
File "/src/ngsi-timeseries-api/src/translators/crate.py", line 157, in _store_medatata
self.cursor.execute(stmt, (table_name, persisted_metadata))
File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 53, in execute
self._result = self.connection.client.sql(sql, parameters,
File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 331, in sql
content = self._json_request('POST', self.path, data=data)
File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 458, in _json_request
_raise_for_status(response)
File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 187, in _raise_for_status
raise ProgrammingError(error.get('message', ''),
crate.client.exceptions.ProgrammingError: SQLActionException[InvalidColumnNameException: "5.9" contains a dot]
DEBUG:urllib3.connectionpool:http://crate-db:4200 "POST /_sql?error_trace=true HTTP/1.1" 400 6596
DEBUG:connexion.apis.abstract:Getting data and status code
DEBUG:connexion.apis.abstract:Prepared body and status code (500)
DEBUG:connexion.apis.abstract:Getting data and status code
DEBUG:connexion.apis.abstract:Got framework response
The orion logs shows the following:
logInfoRequestWithPayload | msg=Request received: POST /v2/entities/station:2/attrs?type=WeatherObserved, request payload (338 bytes): {"temperature":{"type":"Integer","value":19,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-03-08T10:23:34.295Z"}}},"location":{"type":"geo:point","value":"36.59389,10.30212","metadata":{"TimeInstant":{"type":"DateTime","value":"2021-03-08T10:23:34.295Z"}}},"TimeInstant":{"type":"DateTime","value":"2021-03-08T10:23:34.295Z"}}, response code: 204
time=2021-03-08T10:23:34.327Z | lvl=WARN | corr=56645b84-7ff8-11eb-b242-0242ac120006; cbnotif=1 | trans=1615196386-384-00000000093 | from=172.18.0.8 | srv=smartwater | subsrv=/weatherstation | comp=Orion | op=httpRequestSend.cpp[583]:httpRequestSendWithCurl | msg=Notification response NOT OK, http code: 500
time=2021-03-08T10:23:34.327Z | lvl=INFO | corr=56645b84-7ff8-11eb-b242-0242ac120006; cbnotif=1 | trans=1615196386-384-00000000093 | from=172.18.0.8 | srv=smartwater | subsrv=/weatherstation | comp=Orion | op=logTracing.cpp[63]:logInfoNotification | msg=Notif delivered (subId: 601aabeb96fce3b0209b8aa8): POST quantumleap:8668/v2/notify, response code: 500
time=2021-03-08T10:23:34.327Z | lvl=WARN | corr=56645b84-7ff8-11eb-b242-0242ac120006; cbnotif=2 | trans=1615196386-384-00000000094 | from=172.18.0.8 | srv=smartwater | subsrv=/weatherstation | comp=Orion | op=httpRequestSend.cpp[583]:httpRequestSendWithCurl | msg=Notification response NOT OK, http code: 500
Does anyone have an idea what's the problem here?
Thank you!

The error is clearly an attribute name as reported by the message:
InvalidColumnNameException: "5.9" contains a dot
Try using 5_9 as attribute name.

Related

AWS NoSQL Workbench - ValueError: 'localhost' is not a valid AWS region name for dynamodb

I use AWS NoSQL Workbench to view my DynamoDBLocal tables.
I can connect to my DynamoDBLocal and list the name of the tables I have, however when I want to see the content of any tables I get this stacktrace :
2022-08-11T08:22:12.749 ERROR --- [ asgi_gw_2] l.aws.handlers.logging : exception during call chain
Traceback (most recent call last):
File "/opt/code/localstack/localstack/aws/chain.py", line 57, in handle
handler(self, self.context, response)
File "/opt/code/localstack/localstack/aws/handlers/service.py", line 121, in __call__
handler(chain, context, response)
File "/opt/code/localstack/localstack/aws/handlers/legacy.py", line 81, in __call__
result = self.forward_request(
File "/opt/code/localstack/localstack/aws/handlers/legacy.py", line 141, in forward_request
return do_forward_request(
File "/opt/code/localstack/localstack/services/edge.py", line 225, in do_forward_request
result = do_forward_request_inmem(api, method, path, data, headers, port=port)
File "/opt/code/localstack/localstack/services/edge.py", line 249, in do_forward_request_inmem
response = modify_and_forward(
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 604, in wrapper
value = func(*args, **kwargs)
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 684, in modify_and_forward
listener_result = listener.forward_request(
File "/opt/code/localstack/localstack/http/adapters.py", line 51, in forward_request
response = self.request(request)
File "/opt/code/localstack/localstack/aws/proxy.py", line 46, in request
return self.skeleton.invoke(context)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
return self.dispatch_request(context, instance)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
result = handler(context, instance) or {}
File "/opt/code/localstack/localstack/aws/forwarder.py", line 55, in _call
return handler(context, req)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
return self.fn(*args, **kwargs)
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 439, in describe_table
table_props = get_store(context).table_properties.get(table_name)
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 295, in get_store
return dynamodb_stores[_account_id][_region]
File "/opt/code/localstack/localstack/services/stores.py", line 162, in __getitem__
raise ValueError(
ValueError: 'localhost' is not a valid AWS region name for dynamodb
2022-08-11T08:22:12.749 INFO --- [ asgi_gw_2] localstack.request.aws : AWS dynamodb.DescribeTable => 500 (InternalError)
But when I use the cli it works :
aws dynamodb scan --table-name my_table --endpoint-url http://localhost:my_port

sqlalchemy Object with OrderingList - Session.commit() method fails

Model that has attribute in type of sqlalchemy OrderingList fails on Database Session Commit step (as Database I am using PostgreSQL 13):
from sqlalchemy import Column, Integer, String
from sqlalchemy.orm import relationship
from database.base_class import Base
from sqlalchemy.ext.orderinglist import ordering_list
class ContentType(Enum):
SCENE_HEADING = 'scene_heading'
ACTION = 'action'
CHARACTER = 'character'
PARENTHETICAL = 'parenthetical'
DIALOGUE = 'dialogue'
SHOT = 'shot'
TRANSITION = 'transition'
TEXT = 'text'
class EditorElement(Base):
"""
Class that represents the Table of Editor Elements.
"""
id = Column(Integer, primary_key=True, autoincrement=True, index=True)
content = Column(String, nullable=True)
content_type = Column(ContentType, nullable=True)
screenplay_id = Column(Integer, ForeignKey('screenplays.id', ondelete="CASCADE"), nullable=False)
screenplay = relationship("Screenplay", back_populates="elements")
class Screenplay(Base):
"""
Class that represents the screenplay table in the database.
"""
id = Column(Integer, primary_key=True, autoincrement=True, index=True)
name = Column(String, index=True, nullable=False)
elements = relationship("EditorElement", back_populates="screenplay", order_by="EditorElement.content_type",
collection_class=ordering_list('content_type'))
I am using FastAPI framework and in the CRUD file where the Screenplay is created, I have the following class:
class CRUDScreenplay(CRUDBase[models.Screenplay, schemas.ScreenplayBase, schemas.ScreenplayBase]):
#staticmethod
def create(db: Session, *, obj_in: schemas.ScreenplayCreate) -> models.Screenplay:
db_obj = models.Screenplay()
db_obj.name = obj_in['name']
for element in obj_in.get('elements'):
e = models.EditorElement(
content=element.content,
content_type=element.content_type,
screenplay_id=db_obj.id
)
db_obj.elements.append(e)
db.add(db_obj)
db.commit()
db.refresh(db_obj)
return db_obj
Using the PyCharm debugger I was able to check when the error occurs, it seems to appear after Session.commit(). Here is the object elements before the commit() method
And here is the elements object after the commit() method.
And the error in the console:
INFO: 127.0.0.1:52649 - "POST /api/v1/screenplays/ HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\sql\sqltypes.py", line 1649, in _object_value_for_elem
return self._object_lookup[elem]
KeyError: 'scene_heading'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 372, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\fastapi\applications.py", line 261, in __call__
await super().__call__(scope, receive, send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
raise exc
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\middleware\cors.py", line 84, in __call__
await self.app(scope, receive, send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\exceptions.py", line 82, in __call__
raise exc
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
raise e
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\routing.py", line 656, in __call__
await route.handle(scope, receive, send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\routing.py", line 259, in handle
await self.app(scope, receive, send)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\routing.py", line 61, in app
response = await func(request)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\fastapi\routing.py", line 235, in app
response_data = await serialize_response(
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\fastapi\routing.py", line 130, in serialize_response
value, errors_ = await run_in_threadpool(
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\starlette\concurrency.py", line 39, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\anyio\to_thread.py", line 28, in run_sync
return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\anyio\_backends\_asyncio.py", line 818, in run_sync_in_worker_thread
return await future
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\anyio\_backends\_asyncio.py", line 754, in run
result = context.run(func, *args)
File "pydantic\fields.py", line 854, in pydantic.fields.ModelField.validate
File "pydantic\fields.py", line 1071, in pydantic.fields.ModelField._validate_singleton
File "pydantic\fields.py", line 1118, in pydantic.fields.ModelField._apply_validators
File "pydantic\class_validators.py", line 313, in pydantic.class_validators._generic_validator_basic.lambda12
File "pydantic\main.py", line 678, in pydantic.main.BaseModel.validate
File "pydantic\main.py", line 562, in pydantic.main.BaseModel.from_orm
File "pydantic\main.py", line 1001, in pydantic.main.validate_model
File "pydantic\utils.py", line 409, in pydantic.utils.GetterDict.get
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\orm\attributes.py", line 481, in __get__
return self.impl.get(state, dict_)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\orm\attributes.py", line 941, in get
value = self._fire_loader_callables(state, key, passive)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\orm\attributes.py", line 977, in _fire_loader_callables
return self.callable_(state, passive)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\orm\strategies.py", line 911, in _load_for_state
return self._emit_lazyload(
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\orm\strategies.py", line 1051, in _emit_lazyload
result = result.unique().scalars().all()
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\engine\result.py", line 1371, in all
return self._allrows()
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\engine\result.py", line 401, in _allrows
rows = self._fetchall_impl()
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\engine\result.py", line 1284, in _fetchall_impl
return self._real_result._fetchall_impl()
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\engine\result.py", line 1696, in _fetchall_impl
return list(self.iterator)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\orm\loading.py", line 147, in chunks
fetch = cursor._raw_all_rows()
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\engine\result.py", line 393, in _raw_all_rows
return [make_row(row) for row in rows]
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\engine\result.py", line 393, in <listcomp>
return [make_row(row) for row in rows]
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\sql\sqltypes.py", line 1768, in process
value = self._object_value_for_elem(value)
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\sql\sqltypes.py", line 1651, in _object_value_for_elem
util.raise_(
File "C:\Users\userh\Playground\screenplay-writer\backend\.venv\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
LookupError: 'scene_heading' is not among the defined enum values. Enum name: None. Possible values: None
However, the data was successfully created in the database, but FastAPI does not return anything due to this error.
There are some related issues in GitHub but I am was not able to figure out what is going on there:
Issue 1
Issue 2
Issue 3
Issue 4
UPDATE
Alembic (which is often used with FastAPI) or SLQAlchemy created the PostgreSQL ENUM Type in Uppercase.
I changed the definition of ContentType(enum.Enum) as it was in the example of the official documentation LINK. Thanks to #fchancel for pointing on that:
class ContentType(str, enum.Enum):
HEADING = 'HEADING'
ACTION = 'ACTION'
CHARACTER = 'CHARACTER'
PARENTHETICAL = 'PARENTHETICAL'
DIALOGUE = 'DIALOGUE'
SHOT = 'SHOT'
TRANSITION = 'TRANSITION'
TEXT = 'TEXT'
So, now my code works
It seems that when creating your elements, the content_type section is a problem, since it can't find heading_scene. To understand this, we need to see how you call your function and what obj_in contains.
However, it's possible that the source of the error is just your class ContentType(Enum) which to work properly should be class ContentType(str, Enum)

How do I fix a TypeError: Object of type bytes is not JSON serializable that is only manifesting inside Docker container?

I am trying to use connexion/Flask to return a base64-encoded image generated by PIL/pillow in a HTTP Response.
Having upgraded to python3.7 from python2.7, that part of the code works unaltered.
However, when running inside Docker (python3.7-slim), I get the time-honoured TypeError: Object of type bytes is not JSON serializable (full Traceback below).
The encoding is undertaken via
cutout_b64 = base64.b64encode(sbuffer.getvalue()).decode("ascii")
PLEASE NOTE: There are many many many answers that suggest applying .decode(...) but this does not work inside Docker! The question is:
Q. Why does this work outside Docker but not inside the container? How do I fix this? Is it a LANG difference perhaps?
Thanks as ever
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/site-packages/connexion/decorators/decorator.py", line 68, in wrapper
response = function(request)
File "/usr/local/lib/python3.7/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
response = function(request)
File "/app/custom_validators.py", line 107, in wrapper
response = function(request)
File "/usr/local/lib/python3.7/site-packages/connexion/decorators/response.py", line 110, in wrapper
return _wrapper(request, response)
File "/usr/local/lib/python3.7/site-packages/connexion/decorators/response.py", line 90, in _wrapper
self.operation.api.get_connexion_response(response, self.mimetype)
File "/usr/local/lib/python3.7/site-packages/connexion/apis/abstract.py", line 365, in get_connexion_response
response = cls._response_from_handler(response, mimetype)
File "/usr/local/lib/python3.7/site-packages/connexion/apis/abstract.py", line 345, in _response_from_handler
return cls._build_response(mimetype=mimetype, data=response, extra_context=extra_context)
File "/usr/local/lib/python3.7/site-packages/connexion/apis/flask_api.py", line 186, in _build_response
data, status_code, serialized_mimetype = cls._prepare_body_and_status_code(data=data, mimetype=mimetype, status_code=status_code, extra_context=extra_context)
File "/usr/local/lib/python3.7/site-packages/connexion/apis/abstract.py", line 417, in _prepare_body_and_status_code
body, mimetype = cls._serialize_data(data, mimetype)
File "/usr/local/lib/python3.7/site-packages/connexion/apis/flask_api.py", line 203, in _serialize_data
body = cls.jsonifier.dumps(data)
File "/usr/local/lib/python3.7/site-packages/connexion/jsonifier.py", line 57, in dumps
return self.json.dumps(data, **kwargs) + '\n'
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 211, in dumps
rv = _json.dumps(obj, **kwargs)
File "/usr/local/lib/python3.7/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 201, in encode
chunks = list(chunks)
File "/usr/local/lib/python3.7/json/encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 438, in _iterencode
o = _default(o)
File "/usr/local/lib/python3.7/site-packages/connexion/apps/flask_app.py", line 146, in default
return json.JSONEncoder.default(self, o)
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 100, in default
return _json.JSONEncoder.default(self, o)
File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

Json loads fails on mqtt message

I'm trying to parse the messages sent on an MQTT topic as JSONs, but when i try to call json.loads() on the decoded string, the code goes into exception:
File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 153 (char 152)
The code is the following:
def on_message_mqtt(self, client, userdata, message):
m_decode = message.payload.decode('utf-8')
logger.info(m_decode)
m_in = json.loads(m_decode)
logger.info(m_in)
While the payload of the messages has the following format:
{"Timestamp":"2021-05-24-13:27:13.450","AI":[1204,7,0,4,18,2,1176,802,11,0,381,2496,0,0,810,1282],"DI":[false,false,false,false,false,false,false,false]}
I tried to put the text into a string and loads() is able to parse it, so I bet that the problem is on the initial string decoding.
Here you can see a print of the decoded string, and then the error on the call to json.loads():
2021-05-24 16:31:47 - INFO - {"Timestamp":"2021-05-24-14:31:44.790","AI":[912,9,0,2,16,2,886,605,11,0,321,1924,1,963,620,9],"DI":[false,false,false,false,false,false,false,false]}
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "router.py", line 136, in run
self.mqtt.loop_forever()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 1779, in loop_forever
rc = self.loop(timeout, max_packets)
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 1181, in loop
rc = self.loop_read(max_packets)
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 1572, in loop_read
rc = self._packet_read()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 2310, in _packet_read
rc = self._packet_handle()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 2940, in _packet_handle
return self._handle_pubrel()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 3246, in _handle_pubrel
self._handle_on_message(message)
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 3444, in _handle_on_message
self.on_message(self, self._userdata, message)
File "router.py", line 87, in on_message_mqtt
logger.info(json.loads(m_decode), exc_info=True)
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 151 (char 150)
Judging from the error, character 152 is the square bracket closing the arroy of the value for the key of "DI" ] which would imply incorrect parsing using the payload.decode function
Can you check if this works
def on_message_mqtt(self, client, userdata, message):
m_in = json.loads(message.decode("utf-8"))
logger.info(m_in)

Intermittent IndexError with MySQL and Django running on Ubuntu

Here is what I'm getting:
Traceback (most recent call last):
File "/.../.env/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/.../.env/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/.../.env/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/.../.env/lib/python3.6/site-packages/django/contrib/auth/decorators.py", line 20, in _wrapped_view
if test_func(request.user):
File "/.../.env/lib/python3.6/site-packages/django/contrib/auth/decorators.py", line 44, in <lambda>
lambda u: u.is_authenticated,
File "/.../.env/lib/python3.6/site-packages/django/utils/functional.py", line 213, in inner
self._setup()
File "/.../.env/lib/python3.6/site-packages/django/utils/functional.py", line 347, in _setup
self._wrapped = self._setupfunc()
File "/.../.env/lib/python3.6/site-packages/django/contrib/auth/middleware.py", line 24, in <lambda>
request.user = SimpleLazyObject(lambda: get_user(request))
File "/.../.env/lib/python3.6/site-packages/django/contrib/auth/middleware.py", line 12, in get_user
request._cached_user = auth.get_user(request)
File "/.../.env/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 189, in get_user
user = backend.get_user(user_id)
File "/.../.env/lib/python3.6/site-packages/django/contrib/auth/backends.py", line 98, in get_user
user = UserModel._default_manager.get(pk=user_id)
File "/.../.env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/.../.env/lib/python3.6/site-packages/django/db/models/query.py", line 393, in get
num = len(clone)
File "/.../.env/lib/python3.6/site-packages/django/db/models/query.py", line 250, in __len__
self._fetch_all()
File "/.../.env/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/.../.env/lib/python3.6/site-packages/django/db/models/query.py", line 63, in __iter__
for row in compiler.results_iter(results):
File "/.../.env/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1007, in apply_converters
value = row[pos]
IndexError: list index out of range
As you can see, there is none of my code in the stacktrace, all I know is that this code is happening somewhere in a very simple view that gets a queryset and renders it to JSON.
Most of the times it works, sometimes this traceback is thrown and the page 500s.
What the hell is going on?
Sometimes I get other weird errors:
unsupported operand type(s) for +=: 'int' and 'str'
triggered by
count += Model.objects.filter(...).count()
I've tried to replicate from a shell, but that's not working.