Currently AWS QuickSight has an option to refresh the Dataset from S3 using scheduled refresh.
Once the data is refreshed, normal dashboards (visuals chart) gets updated with new data.
For Anomaly dashboards (insights chart), currently if the data is refreshed, we need to manually trigger the "run now" and later run "update".
Is their a way or option to automate the refresh for Anomaly dashboard as well? Can we trigger it using programming?
I used AWS CLI with below command:
aws quicksight create-ingestion --aws-account-id <YOUR-AWS-ACCOUNT --ingestion-id <UNIQUE-ID> --data-set-id <DATASET-ID> --region <REGION>
--aws-accout-id -> The account to your desired quicksight
--ingestion-id -> This can be any unique name(string)
--data-set-id -> Go to QuckSight dataset to find the unique string
--region -> Your region for the aws-account
Also I did the same using Python boto3:
import boto3
import calendar
import time
client = boto3.client(
'quicksight',
region_name='us-east-1'
)
#to create unique number
ingestionID = calendar.timegm(time.gmtime())
response = client.create_ingestion(
DataSetId='12345e-d123-4123-b1a1-12f30154b123',
IngestionId=str(ingestionID),
AwsAccountId='123456789123'
)
print(response)
Related
I need to import the existing Aurora cluster in to terraform. I tried terraform import aws_rds_cluster.sample_cluster cluster statement.
I got the state file ready also I could also do Terraform show However, When I try to destroy the cluster Terraform tries to delete the cluster without the instances under it -
So the destroy command is failing.
`Error: error deleting RDS Cluster (test): InvalidDBClusterStateFault: Cluster cannot be deleted, it still contains DB instances in non-deleting state.status code: 400, request id: 15dfbae8-aa13-4838-bc42-8020a2c87fe9`
Is there a way I can import the entire cluster that includes instances as well? I need to have a single statefile that can be used to manage entire cluster(including underlying instances).
Here is the main.tf that is getting used to call the import -
access_key = "***"
secret_key = "*****"
region = "us-east-1"
}
resource "aws_rds_cluster" "test" {
engine = "aurora-postgresql"
engine_version = "11.9"
instance_class = "db.r5.2xlarge"
name = "test"
username = "user"
password = "******"
parameter_group_name = "test"
}```
Based on the comments.
Importing just aws_rds_cluster into TF is not enough. One must also import all aws_rds_cluster_instance resources which are part of the cluster.
If the existing infrastructure is complex, instead of fully manual development of TF config files for the importing procedure, an open-sourced third party tool, called former2, could be considered. The tool can generate TF config files from existing resources:
Former2 allows you to generate Infrastructure-as-Code outputs from your existing resources within your AWS account.
TF is one of the outputs supported.
I'm able to track the file ingress in a single bucket using below code. But I want to track all the files going into different buckets of my project on Google Cloud..is there any way?
def hello_gcs(event, context):
"""Triggered by a change to a Cloud Storage bucket.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
import os
from google.cloud import bigquery
BQ = bigquery.Client()
table_id = 'xx.DW_STAGE.Bucket_Monitor'
table = BQ.get_table(table_id)
bucket=event['bucket']
file_nm=event['name']
create_ts=event['timeCreated']
update_ts=event['updated']
size=event['size']
contentType=event['contentType']
crc32c=event['crc32c']
etag=event['etag']
generation=event['generation']
file_id=event['id']
kind=event['kind']
md5Hash=event['md5Hash']
medialink=event['mediaLink']
metageneration=event['metageneration']
selfLink=event['selfLink']
storageClass=event['storageClass']
timeStorageClassUpdated=event['timeStorageClassUpdated']
errors = BQ.insert_rows(table, [(bucket,file_nm,create_ts,update_ts,size,contentType,crc32c,etag,generation,file_id,kind,md5Hash,medialink,metageneration,selfLink,storageClass,timeStorageClassUpdated)]) # Make an API request.
if errors == []:
print("New rows have been added.")
else:
print("Encountered errors while inserting rows: {}".format(errors))
Maybe you will be interested in Pub/Sub. This mechanism is used underneath by Cloud Function Storage triggers (description is here).
For every of your buckets you can create Pub/Sub notification in the same topic (How to create notification you may find here). Every notification can has attributes and payload which contain information about the object which changed.
Such notification can be used as trigger for Cloud Function similar to yours.
I need to get jenkins all master/child job details like start time, end time, review id which got integrated in current jenkin job for all the master and children jobs and put the same in a json array. I have written below script to get the details but not able to get start and end time of the job for master/child and how do we differentiate between master job detail and child job detail data.
import hudson.model.*
map1=[:]
map1["BUILD_NUMBER"]=(build.getEnvVars()['BUILD_NUMBER'])
map1["PARENT_BUILD_NUMBER"]=(build.getEnvVars()['BUILD_NUMBER'])
map1["JOB_NAME"]=(build.getEnvVars()['JOB_NAME'])
map1["PHASE_NAME"]=(build.getEnvVars()['JOB_NAME'])
map1["Status"]=(build.getEnvVars()['PHASE_RESULT'])
map1["Branch"]="master"
map1["JOB_URL"]=(build.getEnvVars()['JOB_URL'])
map1["START_TIME"]=(build.getEnvVars()['timestamp'])
map1["END_TIME"]=(build.getEnvVars()['buildEndTime'])
def json = new groovy.json.JsonBuilder()
json MasterJob: map1
println "json output: "
println groovy.json.JsonOutput.prettyPrint(json.toString())
Jenkin job image
You can get that information using Jenkins' REST API. It provides build information in various formats (including JSON). Example url:
http://JENKINS_HOST:8080/job/JOB_NAME/lastSuccessfulBuild/api/json?pretty=true
You have also a link to REST API documentation in bottom right corner of your Jenkins instance.
Let's say I need to create a lot of different documents/collections in firestore. I need to add it quickly, like copy and paste json. I can't do that with standard firebase console, because adding 100 documents will take me forever. Is there any solutions for to bulk create mock data with a given structure in firestore db?
If you switch to the Cloud Console (rather than Firebase Console) for your project, you can use Cloud Shell as a starting point.
From the Cloud Shell environment you'll find tools like node and python installed and available. Using whatever one you prefer, you can write a script using the Server Client libraries.
For example in Python:
from google.cloud import firestore
import random
MAX_DOCUMENTS = 100
SAMPLE_COLLECTION_ID = u'users'
SAMPLE_COLORS = [u'Blue', u'Red', u'Green', u'Yellow', u'White', u'Black']
# Project ID is determined by the GCLOUD_PROJECT environment variable
db = firestore.Client()
collection_ref = db.collection(SAMPLE_COLLECTION_ID)
for x in range(0, MAX_DOCUMENTS - 1):
collection_ref.add({
u'primary': random.choice(SAMPLE_COLORS),
u'secondary': random.choice(SAMPLE_COLORS),
u'trim': random.choice(SAMPLE_COLORS),
u'accent': random.choice(SAMPLE_COLORS)
})
While this is the easiest way to get up and running with a static dataset, it lives a little to be desired. Namely with Firestore, live dynamic data is needed to exercises it's functionally, such as real-time queries. For this task, using Cloud Scheduler & Cloud Functions is a relatively easy way to regularly updating sample data.
In addition to the sample generation code, you'll specific the update frequency in the Cloud Scheduler. For instance in the image below, */10 * * * * defines a frequency of every 10 minutes using the standard unix-cron format:
For non-static data, often a timestamp is useful. Firestore provides a way to have a timestamp from the database server added at write-time as one of the fields:
u'timestamp': firestore.SERVER_TIMESTAMP
It is worth noting that timestamps like this will hotspot in production systems if not sharded correctly. Typically 500 writes/second to the same collection is the maximum you will want so that the index doesn't hotspot. Sharding can be as simple something like as each user having their own collection (500 writes/second per user). However for this example, writing 100 documents every minute via a scheduled Cloud Function is definitely not an issue.
FireKit is a good resource to use for this purpose. It even allows sub-collections.
https://retroportalstudio.gumroad.com/l/firekit_free
I know how to insert a new group via MySQL, and it works, to a degree. The problem is that the database changes are not loaded into memory if you insert the group manually. Sending a HUP signal to the process does work, but it is kludgy and a hack. I desire elegance :)
What I am looking to do, if possible is to make changes (additions/deletions/changes) to a group via MySQL, and then send an HTTP request to the openfire server to read the new changes. Or in the alternative, add/delete/modify groups similar to how the User Service works.
If anyone can help I would appreciate it.
It seems to me that if sending a HUP signal works for you, then that's actually quite a simple, elegant and efficient way to get Openfire to read your new group, particularly if you do it with the following command on the Openfire server (and assuming it's running a Linux/Unix OS):
pkill -f -HUP openfire
If you still want to send an HTTP request to prompt Openfire to re-read the groups, the following Python script should do the job. It is targeted at Openfire 3.8.2, and depends on Python's mechanize library, which in Ubuntu is installed with the python-mechanize package. The script logs into the Openfire server, pulls up the Cache Summary page, selects the Group and Group Metadata Cache options, enables the submit button and then submits the form to clear those two caches.
#!/usr/bin/python
import mechanize
import cookielib
# Customize to suit your setup
of_host = 'http://openfire.server:9090'
of_user = 'admin_username'
of_pass = 'admin_password'
# Initialize browser and cookie jar
br = mechanize.Browser()
br.set_cookiejar(cookielib.LWPCookieJar())
# Log into Openfire server
br.open(of_host + '/login.jsp')
br.select_form('loginForm')
br.form['username'] = of_user
br.form['password'] = of_pass
br.submit()
# Select which cache items to clear in the Cache Summary page
# On my server, 13 is Group and 14 is Group Metadata Cache
br.open(of_host + '/system-cache.jsp')
br.select_form('cacheForm')
br.form['cacheID'] = ['13','14']
# Activate the submit button and submit the form
c = br.form.find_control('clear')
c.readonly = False
c.disabled = False
r = br.submit()
# Uncomment the following line if you want to view results
#print r.read()