AWS X-Ray Python SDK get_service_graph - json

I am trying to get JSON using get_service_graph() provided by AWS X-Ray Python SDK in AWS Lambda function. reference link
import boto3
from datetime import datetime
def lambda_handler(event, context):
client = boto3.client('xray')
response1 = client.get_service_graph(
StartTime=datetime(2017, 5, 20, 12, 0),
EndTime=datetime(2017, 5, 20, 18, 0)
)
return response1
However, when I passed StartTime and EndTime parameters, stack trace reports datetime type is not JSON serializable. I even tried the following way.
response1 = client.get_service_graph(
StartTime="2017-05-20 00:00:00",
EndTime="2017-05-20 02:00:00"
)
What's weird is, if EndTime is set as "2017-05-20 01:00:00", there is no error generated. Other than that, the same error occurred.
{
"stackTrace": [
[
"/usr/lib64/python2.7/json/__init__.py",
251,
"dumps",
"sort_keys=sort_keys, **kw).encode(obj)"
],
[
"/usr/lib64/python2.7/json/encoder.py",
207,
"encode",
"chunks = self.iterencode(o, _one_shot=True)"
],
[
"/usr/lib64/python2.7/json/encoder.py",
270,
"iterencode",
"return _iterencode(o, 0)"
],
[
"/var/runtime/awslambda/bootstrap.py",
104,
"decimal_serializer",
"raise TypeError(repr(o) + \" is not JSON serializable\")"
]
],
"errorType": "TypeError",
"errorMessage": "datetime.datetime(2017, 5, 20, 1, 53, 13, tzinfo=tzlocal()) is not JSON serializable"
}
I did try only use date, like datetime(2017, 5, 20). However, if I use two consecutive days as StartTime and EndTime, the runtime complains the interval can't be more than 6 hours. If I use same date, it only returns empty JSON. I don't know how to get granularity of get_service_graph().
I think Python SDK for AWS X-Ray might be premature, but I'd still like to seek help from someone who had the same experience. Thanks!

the right way is using datetime(2017, 5, 20) not a string... but can you try using only date... without time? at least the AWS docs shows an example exactly like yours but only yyyy-mm-dd without time

Related

Unable to process JSON input when ingesting logs to Oracle

Error:
send: b'{"specversion": "1.0", "logEntryBatches": [{"entries": [{"data": "{\\"hello\\": \\"oracle\\", \\"as\\": \\"aaa\\"}", "id": "ocid1.test.oc1..jkhjkhh23423fd", "time": "2021-04-01T12:19:28.416000Z"}], "source": "EXAMPLE-source-Value", "type": "remediationLogs", "defaultlogentrytime": "2021-04-01T12:19:28.416000Z"}]}'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Date: Fri, 02 Apr 2021 07:39:16 GMT
header: opc-request-id: ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/770899C2C7CA6ABA11D996CC57E8EE8F
header: Content-Type: application/json
header: Connection: close
header: Content-Length: 79
Traceback (most recent call last):
File "tool.py", line 45, in <module>
put_logs_response = loggingingestion_client.put_logs(
File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/loggingingestion/logging_client.py", line 172, in put_logs
return self.base_client.call_api(
File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/base_client.py", line 276, in call_api
response = self.request(request)
File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/base_client.py", line 388, in request
self.raise_service_error(request, response)
File "/home/ubuntu/.local/lib/python3.8/site-packages/oci/base_client.py", line 553, in raise_service_error
raise exceptions.ServiceError(
oci.exceptions.ServiceError: {'opc-request-id': 'ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/770899C2C7CA6ABA11D996CC57E8EE8F', 'code': 'InvalidParameter', 'message': 'Unable to process JSON input', 'status': 400}
I am trying to send json data to Oracle logs, but getting the above error. I am using json.dumps(data) to convert the dict to string. Kindly let me know if any workaround available to this.
Code:
data = {'hello':'oracle', "as":"aaa"}
put_logs_response = loggingingestion_client.put_logs(
log_id="ocid1.log.oc1.iad.<<Log OCID>>",
put_logs_details=oci.loggingingestion.models.PutLogsDetails(
specversion="1.0",
log_entry_batches=[
oci.loggingingestion.models.LogEntryBatch(
entries=[
oci.loggingingestion.models.LogEntry(
data= json.dumps(data),
id="ocid1.test.oc1..jkhjkhh23423fd",
time=datetime.strptime(
"2021-04-01T12:19:28.416Z",
"%Y-%m-%dT%H:%M:%S.%fZ"))],
source="EXAMPLE-source-Value",
type="Logs",
defaultlogentrytime=datetime.strptime(
"2021-04-01T12:19:28.416Z",
"%Y-%m-%dT%H:%M:%S.%fZ"))]),
timestamp_opc_agent_processing=datetime.strptime(
"2021-04-01T12:19:28.416Z",
"%Y-%m-%dT%H:%M:%S.%fZ"),
opc_agent_version="EXAMPLE-opcAgentVersion-Value",
opc_request_id="ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/")
This exception indicates that you have an InvalidParameter in your JSON input.
oci.exceptions.ServiceError: {'opc-request-id': 'ER6S6HDVTNWUOKCJ7XXZ/OpcRequestIdExample/770899C2C7CA6ABA11D996CC57E8EE8F', 'code': 'InvalidParameter', 'message': 'Unable to process JSON input', 'status': 400}
The InvalidParameter is your timestamp, which is date - 2021-04-01T12:19:28.416Z.
According to Oracle's documentation you need to use a RFC3339-formatted date-time string with milliseconds precision when creating a LogEntry.
This code snippet is from oci-python-sdk - log_entry.py, but it doesn't mention the milliseconds precision like Oracle's documentation.
#time.setter
def time(self, time):
"""
Sets the time of this LogEntry.
Optional. The timestamp associated with the log entry. An RFC3339-formatted date-time string.
If unspecified, defaults to PutLogsDetails.defaultlogentrytime.
:param time: The time of this LogEntry.
:type: datetime
"""
self._time = time
This code create a UTC RFC3339 complaint timestamp with milliseconds precision
from datetime import datetime
from datetime import timezone
current_utc_time_with_offset = datetime.now(timezone.utc).isoformat()
print(current_utc_time_with_offset)
#output
2021-04-06T13:00:52.706040+00:00
current_utc_time_with_timezone = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
print(current_utc_time_with_timezone)
#output
2021-04-06T13:09:10.053432Z
This Stack Overflow question is worth a read:
What's the difference between ISO 8601 and RFC 3339 Date Formats?
This article is also useful:
Understanding about RFC 3339 for Datetime and Timezone Formatting in Software Engineering
Your data looks fine , I think the issue is that Time precision is more than millisec. it should work fine if you loose the trailing zeros in time.
Format a datetime into a string with milliseconds
https://docs.oracle.com/en-us/iaas/api/#/en/logging-dataplane/20200831/LogEntry/
Your time is RFC3339 but precision is more than millisec
'{"specversion": "1.0", "logEntryBatches": [{"entries": [{"data": "{\"hello\": \"oracle\", \"as\": \"aaa\"}", "id": "ocid1.test.oc1..jkhjkhh23423fd", "time": "2021-04-01T12:19:28.416000Z"}], "source": "EXAMPLE-source-Value", "type": "remediationLogs", "defaultlogentrytime": "2021-04-01T12:19:28.416000Z"}]}'
See https://docs.oracle.com/en-us/iaas/api/#/en/logging-dataplane/20200831/LogEntry/
The timestamp associated with the log entry. An RFC3339-formatted date-time string with milliseconds precision. If unspecified, defaults to PutLogsDetails.defaultlogentrytime.

Problem printing json data from python script

I have a python script that should print json data.
This is what I have in my script:
finaldata = {
"date": datetime.datetime.utcnow().isoformat(),
"voltage_mv":emeter["voltage_mv"],
"current_ma":emeter["current_ma"],
"power_mw":emeter["power_mw"] ,
"energy_wh": emeter["total_wh"],
}
print(finaldata)
I am running the script from Node-RED because I need to send the data to a storage account (in json format of course). The Problem is that the data that is being sent looks like this:
{'power_mw': 0, 'date': '2019-04-16T07:12:19.858159', 'energy_wh': 2, 'voltage_mv': 225045, 'current_ma': 20}
when it should look like this in order to be correctly stored in my storage account:
{"power_mw": 0, "date": '2019-04-16T07:12:19.858159', "energy_wh": 2, "voltage_mv": 225045, "current_ma": 20}
(important for later use, since I already get errors in the storage account).
Does anyone know why this is happening and how I can fix it? Thanks in advance
You should use the python json module and dump your python dict into json:
import json
finaldata = {"power_mw": 0, "date": '2019-04-16T07:12:19.858159',
"energy_wh": 2, "voltage_mv": 225045, "current_ma": 20}
print(json.dumps(finaldata))
JSON Reference
For order check linked OrderedDict
or read the OrderedDict collection Reference

How to take any CSV file and convert it to JSON?(with python as a script engine) [Novice user trying to learn NiFi]

1) There is a CSV file containing the following information (the first row is the header):
first,second,third,total
1,4,9,14
7,5,2,14
3,8,7,18
2) I would like to find the sum of individual rows and generate a final file with a modified header. The final file should look like this:
[
{
"first": 1,
"second": 4,
"third": 9,
"total": 14
},
{
"first": 7,
"second": 5,
"third": 2,
"total": 14
},
{
"first": 3,
"second": 8,
"third": 7,
"total": 18
}
]
But it does not work and I am not sure how to fix this. Can anyone provide me an understanding on how to approach this problem?
NiFi flow:
Although i'm not into Python, by just googling around i think this might do it:
import csv
with open("YOURFILE.csv") as f:
reader = csv.DictReader(f)
data = [r for r in reader]
import json
with open('result.json', 'w') as outfile:
json.dump(data, outfile)
You can use Query Record processor and add new property as
total
select first,second,third,first+second+third total from FLOWFILE
Configure the CsvReader controller service with matching avro schema with int as datatype for all the fields and Json Setwriter controller service,Include total field name so that the output from Query Record processor will be all the columns and the sum of the columns as total.
Connect total relationship from Query Record processor for further processing
Refer to these links regarding Query Record and Configure Record Reader/Writer

Get JSON's attribute value in Chatterbot and Django integration

statement.text in chatterbot and Django integration returns
{'text': u'How are you doing?', 'created_at': datetime.datetime(2017, 2, 20, 7, 37, 30, 746345, tzinfo=<UTC>), 'extra_data': {}, 'in_response_to': [{'text': u'Hi', 'occurrence': 3}]}
I want a value of text attribute so that it prints How are you doing?
The chatterbot return the json object(dict) so you can use the dictionary operations like following
[1]: data = {'text': u'How are you doing?', 'created_at': datetime.datetime(2017, 2, 20, 7, 37, 30, 746345, tzinfo=<UTC>), 'extra_data': {}, 'in_response_to': [{'text': u'Hi', 'occurrence': 3}]}
[2]: data['text'] or data.get('text')[this approch is good].
What you got is dictionary. Value of dictionary can be obtained by get() function. You can also use dict['text'], but it does not perform error checking. get function returns None if key is not present.

R: getting google finance JSON data into a dataframe

I am trying to get google finance JSON data into a dataframe.
I tried:
library(jsonlite)
dat1 <- fromJSON("http://www.google.com/finance/info?q=NSE:%20AAPL,MSFT,TSLA,AMZN,IBM")
dat1
However I get an error:
Error in feed_push_parser(readBin(con, raw(), n), reset = TRUE) :
parse error: trailing garbage
Thank you for any help.
I could not replicate your error using fromJSON due to proxy issues from my side but the following works using httr
require(jsonlite)
require(httr)
#Set your proxy setting if needed
#set_config(use_proxy(url='hostname',port= port,username="",password=""))
url.name = "http://www.google.com/finance/info?q=NSE:%20AAPL,MSFT,TSLA,AMZN,IBM"
url.get = GET(url.name)
#parsing the content as json results in similar error as you encountered
#url.content = content(url.get,type="application/json")
#Error in parseJSON(txt) : parse error: trailing garbage
# " : "0.57" ,"yld" : "2.46" } ,{ "id": "358464" ,"t" : "MSFT"
# (right here) ------^
#read content as html text
url.content = content(url.get, as="text")
#remove html tags
clean.text = gsub("<.*?>", "", url.content)
#remove residual text
clean.text = gsub("\\n|\\//","",clean.text)
DF = fromJSON(clean.text)
head(DF[,1:10],5)
# id t e l l_fix l_cur s ltt lt lt_dts
#1 22144 AAPL NASDAQ 92.51 92.51 92.51 1 4:00PM EDT May 11, 4:00PM EDT 2016-05-11T16:00:02Z
#2 358464 MSFT NASDAQ 51.05 51.05 51.05 1 4:00PM EDT May 11, 4:00PM EDT 2016-05-11T16:00:02Z
#3 12607212 TSLA NASDAQ 208.96 208.96 208.96 1 4:00PM EDT May 11, 4:00PM EDT 2016-05-11T16:00:02Z
#4 660463 AMZN NASDAQ 713.23 713.23 713.23 1 4:00PM EDT May 11, 4:00PM EDT 2016-05-11T16:00:02Z
#5 18241 IBM NYSE 148.95 148.95 148.95 2 6:59PM EDT May 11, 6:59PM EDT 2016-05-11T18:59:12Z
I got the below code from here. Let me know if this helps. On a side note, I would also recommend netfonds. Netfonds is the only source I've found that provides intra-day tick level data for both historical prices and the open book. I posted some additional links below for pulling the Netfonds data if you're interested.
http://www.blackarbs.com/blog/3/22/2015/how-to-get-free-intraday-stock-data-from-netfonds
http://www.onestepremoved.com/free-stock-data/
import urllib
from datetime import date, datetime
""" googlefinance
This module provides a Python API for retrieving stock data from Google Finance.
"""
_month_dict = {
'Jan': 1,
'Feb': 2,
'Mar': 3,
'Apr': 4,
'May': 5,
'Jun': 6,
'Jul': 7,
'Aug': 8,
'Sep': 9,
'Oct': 10,
'Nov': 11,
'Dec': 12}
# Google doesn't like Python's user agent...
class FirefoxOpener(urllib.FancyURLopener):
version = 'Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11'
def __request(symbol):
url = 'http://google.com/finance/historical?q=%s&output=csv' % symbol
opener = FirefoxOpener()
return opener.open(url).read().strip().strip('"')
def get_historical_prices(symbol, start_date=None, end_date=None):
"""
Get historical prices for the given ticker symbol.
Returns a nested list. fields are Date, Open, High, Low, Close, Volume.
"""
price_data = [data.split(',') for data in __request(symbol).split('\n')[1:]]
for quote in price_data:
quote[0] = _format_date(quote[0])
return price_data
def _format_date(datestr):
""" Change datestr from google format ('20-Jul-12') to the format yahoo uses ('2012-07-20')
"""
parts = datestr.split('-')
day = int(parts[0])
month = _month_dict[parts[1]]
year = int('20'+ parts[2])
return date(year, month, day).strftime('%Y-%m-%d')
If the Google finance endpoint returns a newline delimited json, the solution in R should be:
library(jsonlite)
dat1 <- stream_in(url("http://www.google.com/finance/info?q=NSE:%20AAPL,MSFT,TSLA,AMZN,IBM"))
But it seems the endpoint is not accepting such request (any more?):
HTTP status was '403 Forbidden'