I make weather station which uploads data to my MySQL database every 30-60 minutes. How to make on example temperature plot for a week on my website? I've looked for such option in Highcharts but I don't know does it is possible. Date and time is saved in database as timestamp.
They have an example specifically for time data with irregular intervals: http://www.highcharts.com/demo/spline-irregular-time
Get your data from database for last week, then preprocess in backend to fit Highcharts data format, and as result you should have something like this:
var myData = [
[1388534400000, 12],
[next_timestamp, next_value],
[another_timestamp, another_value],
...
]
Now you can use that data to generate chart:
$("#container").highcharts({
series: [{
data: myData
}]
})
Note: timestamps are in milliseconds.
Now to update chart every 30minutes, just create call some AJAX call to get new data from the server:
setInterval(function() {
$.getJSON('path/to/data', function(myData) {
$("#container").highcharts().series[0].setData(myData);
});
}, 30 * 60 * 1000); // 30minutes
Related
How to delete Mongodb documents based on certain date condition using PyMongo which stores the date object in string format in MongoDB ?
Sample code:
API takes a date parameter as payload and returns only that much data.So before inserting new documents, want to ensure previous data is not delete and only the ones which are being currently processed.
import requests
import json
from pymongo import MongoClient
from datetime import datetime, timedelta
from dateutil.relativedelta import relativedelta
past_4_month= datetime.today() + relativedelta(months=-4)
past_4_month_1st_day=past_4_month.strftime("%m/01/%Y")
url='xxx'
header={'Auth':'xyz', 'Content-Type': 'application/json'}
payload={'start_date': past_4_month_1st_day }
api_data = requests.request("POST", url, data=json.dumps(payload), headers=header).json()
client = MongoClient('mongodb://localhost:27017')
db = client[test_db][test_collection]
db.delete_many({'date':{'$gte':past_4_month_1st_day}})
db.insert_many(api_data)
client.close()
Sample output stored in mongodb:
{
"id": ObjectId("5f44e33ffb3be4ed19a10ed3"),
"xyz_id" : "12345",
"name" : "xyz",
"date" : "3/3/2020"
}
For example - when running in Aug-2020 - it will return data from past 4 months i.e. May-2020 onwards and always truncate/delete and load same data until end of Aug-2020.
On Sep-2020 1st of the month, the prev 4 month date would be June-2020, so it should not delete May-2020 data but only from June-2020 till current month (Sep-2020). Meaning only process past 4 months data always with every run (truncate/delete and reload).
However, since the date is stored as string in MongoDB document, tried using converting it to date from string using $dateFromString, $toDate and in Python etc. it is still not working as expected and deletes all the documents (from May-2020) from previous runs/loads.
Please suggest.
I have this API Endpoint setup on Lambda where my applications talk to and get the data it needs.
Problem am running across right now is trying to access an element that is based on the day before today's date.
Language: Python 3.7
Service: AWS Lambda
Provider: WeatherStack
Example: https://weatherstack.com/documentation -> API Features
-> Weather Forecast
In order to access this element from the API provider; I have to basically setup a JSON structure that goes like this:
"forecast": {
"2020-04-04": {
"date": "2020-04-04",
"date_epoch": 1585958400,
"astro": {
"sunrise": "06:42 AM",
"sunset": "07:31 PM",
"moonrise": "03:26 PM",
"moonset": "04:56 AM",
"moon_phase": "Waxing Gibbous",
"moon_illumination": 79
},
"mintemp": 46,
"maxtemp": 54,
"avgtemp": 50,
"totalsnow": 0,
"sunhour": 7.7,
"uv_index": 2
}
}
Now the problem here is the "2020-04-04" date; as I can't access it simply by calling api_endpoint['forecast'][0] as it will throw an error. I checked using Lens however and did find that it does have one element in 'forecast' which is of course the 2020-04-04 that I'm having trouble trying to access.
I don't know if there's a way to dynamically set the element to be called based on yesterday's date since the api provider will change the forecast date element daily.
I've tried api_endpoint['forecast'][datetime.now()] and got an error.
Is there a way to set the [] after ['forecast] dynamically via variable so that i can always call it based on api_endpoint['forecast'][yesterdaysdate]?
Solution:
from datetime import timedelta, datetime
ts = time.gmtime()
todaysdate = (time.strftime("%Y-%m-%d", ts))
yesterday_date = (datetime.datetime.utcnow() - timedelta(1)).strftime('%Y-%m-%d')
data = api_response['forecast'][yesterday_date]
If I understand corectly, you want to call the data inside the api_endpoint['forecast'][yesterday_date].
If so, this can be achieved by this:
from datetime import datetime, timedelta
yesterday_date = (datetime.now() - timedelta(1)).strftime('%Y-%m-%d')
# call to api
api_endpoint['forecast'][yesterday_date]
If you want to days ago, change timedelta(2) and so on.
Today variable can be assigned by this:
current_date = datetime.now().strftime('%Y-%m-%d')
api_endpoint['forecast'][current_date]
If none of the above solutions answer to your question, leave a comment.
I want to write data to CloudWatch using the AWS-SDK (or whatever may work).
I see this:
The only method that looks remotely like publishing data to CloudWatch is the putMetricData method..but it's hard to find an example of using this.
Does anyone know how to publish data to CloudWatch?
When I call this:
cw.putMetricData({
Namespace: 'ec2-memory-usage',
MetricData: [{
MetricName:'first',
Timestamp: new Date()
}]
}, (err, result) => {
console.log({err, result});
});
I get this error:
{ err:
{ InvalidParameterCombination: At least one of the parameters must be specified.
at Request.extractError (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/protocol/query.js:50:29)
at Request.callListeners (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/Users/alex/codes/interos/jenkins-jobs/jobs/check-memory-ec2-instances/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
message: 'At least one of the parameters must be specified.',
code: 'InvalidParameterCombination',
time: 2019-07-08T19:41:41.191Z,
requestId: '688a4ff3-a1b8-11e9-967e-431915ff0070',
statusCode: 400,
retryable: false,
retryDelay: 7.89360948163893 },
result: null }
You're getting this error because you're not specifying any metric data. You're only setting the metric name and the timestamp. You also need to send some values for the metric.
Let's say your application is measuring the latency of requests and you observed 5 requests, with latencies 100ms, 500ms, 200ms, 200ms and 400ms. You have few options for getting this data into CloudWatch (hence the At least one of the parameters must be specified. error).
You can publish these 5 values one at a time by setting the Value within the metric data object. This is the simplest way to do it. CloudWatch does all the aggregation for you and you get percentiles on your metrics. I would not recommended this approach if you need to publish many observations. This option will result in the most requests made to CloudWatch, which may result in a big bill or throttling from CloudWatch side if you start publishing too many observations.
For example:
MetricData: [{
MetricName:'first',
Timestamp: new Date(),
Value: 100
}]
You can aggregate the data yourself and construct and publish the StatisticValues. This is more complex on your end, but results in the fewest requests to CloudWatch. You can aggregate for a minute for example and execute 1 put per metric every minute. This will not give you percentiles (since you're aggregating data on your end, CloudWatch doesn't know the exact values you observed). I would recommend this if you do not need percentiles.
For example:
MetricData: [{
MetricName:'first',
Timestamp: new Date(),
StatisticValues: {
Maximum: 500,
Minimum: 100,
SampleCount: 5,
Sum: 1400
}
}]
You can count the observations and publish Values and Counts. This is kinda the best of both worlds. There is some complexity on your end, but counting is arguably easier than aggregating into StatisticValues. You're still sending every observation so CloudWatch will do the aggregation for you, so you'll get percentiles. The format also allows more data to be sent than in the option 1. I would recommend this if you need percentiles.
For example:
MetricData: [{
MetricName:'first',
Timestamp: new Date(),
Values: [100, 200, 400, 500],
Counts: [1, 2, 1, 1]
}]
See here for more details for each option: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatch.html#putMetricData-property
Background:
We are trying to identify if notifications are due to go out against records that have configurable schedules. So, for instance, a record could have:
NotificationStep: 'week(s)'
NotificationCount: 3
StartDate: 2018-11-17
This would signal that 3 weeks after Nov 17 -- or the last time one of these notifications went out -- we need to send out a new notification.
In SQL this would involve comparing today to subquery (and possibly a union) utilizing the MAX() method.
Question:
Is there a good way to use today's date as the value by which you're comparing your calculated values against?
Something like:
myModel.find({
where:{
new Date(): {$gt: Notifications.max('date', {where: ... }
}
});
All you need is sequelize.where and sequelize.fn
Here you go :
myModel.find({
where:
sequelize.where(sequelize.fn('now') , {$gt: Notifications.max('date', {where: ... } )
});
Note : sequelize.fn('now') // <---- Give you current time from DB
I'm having some troubles working with highcharts.
I want to generate a chart similar to this one : http://www.highcharts.com/demo/column-basic
My problem relates to the series data. I have some SQL queries, one for the Categories (where I fetch distinct data) and other to fetch the results for the series, namely, some X and Y value.
My data from the database has the following format (this is the actual result from the database):
Year Value
2012 9747600000
2012 9358200000
2013 9494830000
2013 9459250000
2013 9478030000
2013 9592300000
2013 9535060000
As far as I can understand, highcharts expects something in this format:
Name: 2012
Data: [9747600000, 9358200000]
Name: 2013
Data: [9747600000, 9358200000]
How can I generate such format directly from MySQL? What's the better way to do this?
Thanks in advance!
EDIT
I thought that the accepted solution worked as I wanted, but I'm still having trouble passing this information to Highcharts. Although the query works as expected, I can't return an array with the following format:
[{
name: 'Tokyo',
data: [49.9, 71.5, 106.4]
}, {
name: 'New York',
data: [83.6, 78.8, 98.5]
}]
Any ideas to solve this problem? Thanks!
You can try query below:
SELECT YR, GROUP_CONCAT(VAL)
FROM TimeTable
GROUP BY YR;
This will get distinct year and values are concatenated as one (comma separated).
Here is the sample SQL Fiddle:
http://sqlfiddle.com/#!2/7859c/1
You can run a foreach loop and put it an array like
`<?php
$newArray = array();
foreach ($result as $year=> $value) {
$newArray[$year][ ] = $value;
}
Print_r($newArray);
?>`