I'm still green in programing and trying to ajust and learn python but I am struggling with reading a csv file and using the content of the file as a value property
I have looked and googled to death and all the solutions puts out
['Deon:app2018:value:1538402685271'] ore a virtical result
example:
session = file content of csv
here is the closes i got
Code:
import urllib.request
import csv
with open('F:\test\session\main\data\credentials\session_id.csv','r') as file:
session_ID = csv.reader(file)
for row in session_ID:
session = "".join(row)
print(session)
url = 'http://webrates.app.com/rates/connect.html?id='+session
print(url)
What i get
Deon:app2018:value:1538402685271
http://webrates.app.com/rates/connect.html?id=
What i want
Deon:app2018:value:1538402685271
http://webrates.app.com/rates/connect.html?id=Deon:app2018:value:1538402685271
Kind Regards
Deon
after lots of trail and error
this solved my problem
import csvlib.request
import csv
with open('F:\test\session\main\data\credentials\session_id.csv','r') as file:
id = file.read() + '\n'
url = 'http://webrates.app.com/rates/connect.html?id='+ id
print(url)
Related
I want to scrape data at the county level from https://apidocs.covidactnow.org
However I could only get a dataframe with one line for each county, and data for each date is stored within a dictionary in each row/county. I would like to access this data and store it in long format (= have one row per county-date).
import requests
import pandas as pd
import os
if __name__ == '__main__':
os.chdir('/home/username/Desktop/')
url = 'https://api.covidactnow.org/v2/counties.timeseries.json?apiKey=ENTER_YOUR_KEY'
response = requests.get(url).json()
data = pd.DataFrame(response)
This seems like a trivial question, but I've tried for hours. What would be the best way to achieve that ?
Do you mean something like that?
import requests
url = 'https://api.covidactnow.org/v2/states.timeseries.csv?apiKey=YOURAPIKEY'
response = requests.get(url)
csv_response = (response.text)
# Then you can transform STRING to CSV
Check this fo string to CSV --> python parsing string to csv format
I've been trying to figure out how to do this for a while without using a framework (I can make this work with Flask for example) but I haven't found anything as of yet. I have two html scripts and a python cgi script. In essence I have the first html file wherein the user enters a string that I read into my python cgi script which in turn does a number of things to finally give me a bunch of strings and json file that I need to pass to another html file and be able to read them there as well.
So far the first half works, and I can open the second html with a redirect which is not elegant but nothing else has worked with the following code:
#!/Users/<username>/opt/anaconda3/bin//python
import pandas as pd
import numpy as np
import cgi
import cgitb
import sys
cgitb.enable()
# Create instance of FieldStorage
form = cgi.FieldStorage()
# Get data from fields
protein_name = form.getvalue('protein_name')
####### function search_results takes in protein_name and gives me the data ###
####### I need to pass to the html file: results.html #########################
if ((search_results(protein_name)!="No protein entered")&(search_results(protein_name)!="No results found")):
all_vars = search_results(protein_name)
##### all_vars is a tuple of strings like gene_name, json files and integers
print("Content-type: text/html","\n\n")
print ('''
<head><meta http-equiv="refresh" content="0;URL='http://localhost/results.html'" /></head>
''')
Any suggestions on how to proceed? Any help is appreciated, thanks!
As you have probably discovered, your current technique issues a redirect to 'results.html', but otherwise discards any results. I don't know exactly what your goals are, but one approach would be to treat 'results.html' as a simple template. Your script will populate it and return it in response to each request. In the example below, 'results.html' can contain arbitrary HTML, along with the line '##RESULTS##', which will be replaced with your output.
#!/Users/<username>/opt/anaconda3/bin/python
import sys
import pandas as pd
import numpy as np
import cgi
import cgitb
cgitb.enable()
def process_results(results):
if results=='No protein entered' or results=='No results found':
return results
# else do something with results, e.g., format into an HTML table
buf = '<table>\n'
for result in results:
buf += f'<tr><td>{cgi.escape(str(result))}</td></tr>\n'
buf += '</table>\n'
return buf
form = cgi.FieldStorage()
protein_name = form.getvalue('protein_name')
results = search_results(protein_name)
print("Content-type: text/html\n")
with open('results.html') as template:
for line in (x.rstrip() for x in template):
if line == '##RESULTS##':
print(process_results(results))
else:
print(line)
Good luck.
I am trying to get as many profile links as I can on khanacademy.org. I am using their api.
I am struggling navigating through the json file to get the desired data.
Here is my code :
from urllib.request import urlopen
import json
with urlopen("https://www.khanacademy.org/api/internal/discussions/video/what-are-algorithms/questions?casing=camel&limit=10&page=0&sort=1&lang=en&_=190422-1711-072ca2269550_1556031278137") as response:
source = response.read()
data= json.loads(source)
for item in data['feedback']:
print(item['authorKaid'])
profile_answers = item['answers']['authorKaid']
print(profile_answers)
My goal is to get as many authorKaid as possible en then store them (to create a database later).
When I run this code I get this error :
TypeError: list indices must be integers or slices, not str
I don't understand why, on this tutorial video : https://www.youtube.com/watch?v=9N6a-VLBa2I at 16:10 it is working.
the issue is item['answers'] are lists and you are trying to access by a string rather than an index value. So when you try to get item['answers']['authorKaid'] there is the error:
What you really want is
print (item['answers'][0]['authorKaid'])
print (item['answers'][1]['authorKaid'])
print (item['answers'][2]['authorKaid'])
etc...
So you're actually wanting to iterate through those lists. Try this:
from urllib.request import urlopen
import json
with urlopen("https://www.khanacademy.org/api/internal/discussions/video/what-are-algorithms/questions?casing=camel&limit=10&page=0&sort=1&lang=en&_=190422-1711-072ca2269550_1556031278137") as response:
source = response.read()
data= json.loads(source)
for item in data['feedback']:
print(item['authorKaid'])
for each in item['answers']:
profile_answers = each['authorKaid']
print(profile_answers)
I'm trying to download and convert this set into a pandas DataFrame structure and display the first 10 lines for viewing in a jupyter notebook.
url = 'https://ckannet-storage.commondatastorage.googleapis.com/2014-12-13T15:15:31.729Z/airfields.json'
resp = requests.get(url)
resp.content
I ran this and it gives me all the content how can I only limit the content so that it can display the first 10 lines only.
Convert your response string to json and then pandas dataframe. Finally select first 10 rows from it. Code -
import pandas as pd
import json
j = json.loads(resp.content)
df = pd.DataFrame(j)
df[:10]
I have fetched some .json data from API.
import urllib2
test=urllib2.urlopen('url')
print test
How can I save it as a table or data frame? I am using Spark 2.0.
This is how I succeeded importing .json data from web into df:
from pyspark.sql import SparkSession, functions as F
from urllib.request import urlopen
spark = SparkSession.builder.getOrCreate()
url = 'https://web.url'
jsonData = urlopen(url).read().decode('utf-8')
rdd = spark.sparkContext.parallelize([jsonData])
df = spark.read.json(rdd)
For this you can have some research and try using sqlContext. Here is Sample code:-
>>> df2 = sqlContext.jsonRDD(test)
>>> df2.first()
Moreover visit line and check for more things here,
https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html
Adding to Rakesh Kumar answer, the way to do it in spark 2.0 is:
http://spark.apache.org/docs/2.1.0/sql-programming-guide.html#data-sources
As an example, the following creates a DataFrame based on the content of a JSON file:
# spark is an existing SparkSession
df = spark.read.json("examples/src/main/resources/people.json")
# Displays the content of the DataFrame to stdout
df.show()
Note that the file that is offered as a json file is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. For more information, please see JSON Lines text format, also called newline-delimited JSON. As a consequence, a regular multi-line JSON file will most often fail.
from pyspark import SparkFiles
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Project").getOrCreate()
zip_url = "https://raw.githubusercontent.com/spark-examples/spark-scala-examples/master/src/main/resources/zipcodes.json"
spark.sparkContext.addFile(zip_url)
zip_df = spark.read.json("file://" +SparkFiles.get("zipcodes.json"))
#click on raw and then copy url