I am making a daily DAG that will run a query in Redshift and will send the result table in an email.
I currently manage to take the query results and put it into a DataFrame. My problem is using send_email (from airflow package) and choosing the parameters for it to show the data.
The output of the email is unfortunately not a table, just the results being all over the place as text.
Here is what I wrote for send_email :
html_content = f"""
<html><body><p>Hello,</p>
<p>Found {num_of_late_dags} Late DAGs:</p>
{late_dags}
<p>Regards,</p>
<p>Me</p>
</body></html>
"""
send_email(to=v_email_recipients_daily_report,
cc=cc,
bcc=bcc,
subject=f'Daily DAG Load Report',
html_content=html_content,
mime_subtype='alternative') ```
Update:
I used pd.to_html() after html_content in .format(to_html()) but that didnt work.
I used to_html() when creating the data as a DataFrame it worked. Not sure why, they should do the same.
So it looked like this: df = pd.DataFrame(data, columns=['X',Y']).to_html()
Then putting df inside the HTML content worked.
Related
I have pulled in data from a number of csv files, as well as a database. I wish to use a merge function to make a dataframe isolating the phone numbers that are contained in both dataframes(one originating from csv, the other originating from the database). However, the dataframe from the database displays as type 'nonetype.' This disallows any operation such as merge. How can i change this to allow the operation?
The data comes in from the database as a list of tuples. I then convert this to a dataframe. However, as stated above, it displays as 'nonetype.' I'm assuming at the moment I am confused about about how dataframes handle data types.
#Grab Data
mycursor = mydb.cursor()
mycursor.execute("SELECT DISTINCT(Cell) FROM crm_data.ap_clients Order By Cell asc;")
apclients = mycursor.fetchall()
#Clean Phone Number Data
for index, row in data.iterrows():
data['phone_number'][index] = data['phone_number'][index][-10:]
for index, row in data2.iterrows():
data2['phone_number'][index] = data2['phone_number'][index][-10:]
for index, row in data3.iterrows():
data3['phone_number'][index] = data3['phone_number'][index][-10:]
#make data frame from csv files
fbl = pd.concat([data,data2,data3], axis=0, sort=False)
#make data frame from apclients(database extraction)
apc = pd.DataFrame(apclients)
#perfrom merge finding all records in both frames
successfulleads= pd.merge(fbl, apc, left_on ='phone_number', right_on='0')
#type(apc) returns NoneType
The expected results are to find all records in both dataframes, along with a count so that I may compare the two sets. Any help is greatly appreciated from this great community :)
So it looks like I had a function to rename the column of the dataframe as shown below:
apc = apc.rename(columns={'0': 'phone_number'}, inplace=True)
for col in apc.columns:
print(col)
the code snippet out of the above responsible:
inplace=True
This snippet dictates whether or not the object is modified in the dataframe, or whether a copy is made. The return type on said object is of nonetype.
Hope this helps whoever ends up in my position. A great thanks again to the community. :)
I am trying to export the output of an 'Analysis of deviance table' in HTML format, so that it can be inserted into a word document.
I created a GLM model as follows:
newmod <- glm(cbind(Recaptured, predated) ~ Morph * Plant * Site, data =
survival, family = binomial)
Running the following code gives me the output that I would like to export to HTML:
anova(newmod,test="Chisq")
I have tried the following code to create a HTML table using stargazer, however it doesn't seem to be working:
anova_mod<-anova(newmod,test="Chisq")
stargazer(newmod, type="html", out = "anova_output.htm")
Is there a simple way of doing this in r? I have managed to successfully export the summary statistics, but what I really need is the Analysis of deviance table.
I believe you are looking for:
print(xtable(anova_mod), type = "html")
as indicated by this answer: Exporting R tables to HTML
Here is my full code for reproducing something similar to your question:
plant.df = PlantGrowth
plant.df$group = factor(plant.df$group,labels = c("Control", "Treatment 1", "Treatment 2"))
newmod = lm(weight ~ group, data = plant.df)
anova_mod=anova(newmod)
anova_mod
install.packages("xtable")
require(xtable)
print(xtable(anova_mod), type = "html")
You can then paste the output to an html vizualizer such as: https://htmledit.squarefree.com/ to see the resulting table.
Instead of printing it, you can write it to a file. I have not personally tested this part, but the second answer in this question should work for you: Save html result to a txt or html file
Note: You can also reference all parts of the anova_mod separately by adding a $ after it like anova_mod$Df.
We have a scenario where we should not display the header in the output in CSV using eText template.
Our output looks like this:
Header000001 Header000002
------------ ------------
Adetail1 Bdetail1
Adetail2 Bdetail2
Adetail3 Bdetail3
Desired output is:
Adetail1 Bdetail1
Adetail2 Bdetail2
Adetail3 Bdetail3
We tried all possible options in eText template like removing header section, verifying the data using BI Publisher Desktop tool, verifying logs etc.
We are not getting any error in BI Publisher Desktop tool.
Same question has been posted by somebody some time ago and it was resolved, but solution was not provided.
It would be very helpful if anybody can provide the exact solution.
The header will just be another block in your eText template. You can use the <DISPLAY CONDITION> command to skip printing that block in the output. The display condition command specifies when the enclosed record or data field group should be displayed. The command parameter is a boolean expression. When it evaluates to true, the record or data field group is displayed. Otherwise the record or data field group is skipped. You can just give condition as false, and that block will be skipped.
I have created a template using the provided data xml to output a CSV, without headers. A delimiter based template is used, but the header is not printed.
Access it from here.
I am looking for a way to convert a rather big (3GB) json file to csv. I tried using R and this is the code that I used.
library("rjson")
data <- fromJSON(file="C:/POI data 30 Rows.json")
json_data <- as.data.frame(data)
write.csv(json_data, file='C:/POI data 30 Rows Exported.csv')
The example I am using is only a subset of the total data of about 30 rows. which I extracted using EMeditor and copied and pasted into a text file. The problem is however it only converts the first row of the data.
I am not an experienced programmer and have tried everything on youtube tutorials from php to excel and nothing seems to work. The problem is I have no Idea what the structure of the data is so I can not create a predetermined data frame and there is a number of missing values within the data.
Any advice would be greatly appreciated.
I have ~10,000 records in a csv file that I need to run through postman and check the output it gives.
I was wondering (because I've been doing this manually) if there's a way to import the csv file into postman, and use the rows like variables and just 'scroll' through each result it gives out.
Cheers,
Daniel
This feature exists in Postman runner.
You have to make sure your CSV column's name(s) match the name(s) of your variable(s) in your collection.
You select your CSV file in the "data" part, just above the "start run", and when you run your collection, it will parse your file and replace the variable(s) with the value in the corresponding column and execute the request.
Then it will loop and get the next value and execute, and so on.
It just depends on the content of your CSV file and how you use it in your Postman request.
You shall have a look at those links :
https://www.getpostman.com/docs/postman/collection_runs/working_with_data_files
http://blog.getpostman.com/2014/10/28/using-csv-and-json-files-in-the-postman-collection-runner/
They will give you complete explanation about how it works.
Alex