I am new to Twitter stream analytics
I was unable to use the tweepy streaming as there was a change in the API version 2.0. So I am currently trying to stream it using a bearer token.
I am facing two issues:
Getting error- The content for this response was already consumed
How to send the JSON response to the Spark stream
I am streaming the JSON response by using stream=True
Any pointers/alternatives would be great!
import requests
import os
import json
bearer_token = 'your bearer token'
query = "fifa"
tweet_fields = "tweet.fields=id,created_at,text"
expansions = "expansions=author_id"
headers = {"Authorization": "Bearer {}".format(bearer_token)}
def create_url(query, tweet_fields, expansions):
url = "https://api.twitter.com/2/tweets/search/recent?query={}&{}".format(
query, tweet_fields, expansions
)
return url
def bearer_oauth(r):
"""
Method required by bearer token authentication.
"""
r.headers["Authorization"] = f"Bearer {bearer_token}"
r.headers["User-Agent"] = "v2SampledStreamPython"
return r
def connect_to_endpoint(url):
response = requests.request("GET", url, auth=bearer_oauth, stream=True)
#print(response.status_code)
for response_line in response.iter_lines():
if response_line:
json_response = json.loads(response_line)
t=json.dumps(json_response, indent=4, sort_keys=True)
if response.status_code != 200:
raise Exception(
"Request returned an error: {} {}".format(
response.status_code, response.text
)
)
def main():
url = create_url(query, tweet_fields, expansions)
timeout = 0
while True:
connect_to_endpoint(url)
timeout += 1
if __name__ == "__main__":
main()
Related
import requests
import random
m_otp=str(random.randint(100000,999999))
print(m_otp)
url = "https://ziper.io/api/send.php?instance_id=My_instance_id&access_token=My_Acess_token&type=json&number=919XXXXXXXXX"
payload = "{\r\n \"text\": \"Hello there \"\r\n }"
headers = {}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
I tried solutions from github but there is the thing I'm getting it.
Do you think of something like:
import requests
import random
import json
m_otp=str(random.randint(100000,999999))
print(m_otp)
url = "https://ziper.io/api/send.php?instance_id=My_instance_id&access_token=My_Acess_token&type=json&number=919XXXXXXXXX"
payload = {"text": "Hello there", "m_otp":m_otp}
payload_str = json.dumps(payload)
headers = {}
response = requests.request("POST", url, headers=headers, data=payload_str)
print(response.text)
?
If not, feel free to tell me the issue and I'll edit my answer.
Also, shouldn't m_otp get passed as a int ?
I've been struggling to solve a problem but could not achieve a solution.
Basically, I'm trying to read JSONs page individually, convert it to a Pandas Dataframe and then store it in a different variable for each time during a for loop.
At the end I would stack each dataframe for create a big dataframe with all the information of each page.
My code:
#Creating a list to store the URLs that will be used to connect
teste = []
for page_num in range(1, 6):
url = "pagina=" + str(page_num) + "&situacao=A&dataadmissaode=&codigoempresa=600"
teste.append(url)
#Loading libraries
import http.client
import pandas as pd
import io
import requests
import json
# auth API
conn = http.client.HTTPSConnection("Company URL")
payload = 'user=MyUser&password=MyPassword'
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
conn.request("GET", "/login", payload, headers)
res = conn.getresponse()
data = res.read()
y = json.loads(data) #load auth and token
access_token = y.get("token", None) #just access token dynamically evey call
#second part: A for routine to execute the code for each URL generated previously in teste list.
my_dictionary = {} #empty dictionary for storing the info if possible
lista_infos = [] #empty list for storing the info if possible
for url in teste:
conn = http.client.HTTPSConnection("my_web_site.com")
payload = url
headers = {
'x-access-token': access_token,
'Content-Type': 'application/x-www-form-urlencoded'
}
conn.request("GET", "/colaborador", payload, headers)
res = conn.getresponse()
data = res.read()
#store in a dataframe and then append it to lista_infos
df_json = pd.read_json(io.StringIO(data.decode('utf-8')))
lista_infos.append(df_json)
Is there another approach to properly store the data and then create a single dataframe with the info from df_json after every call?
I've tried creating a Dataframe of information obtained with a API which does not have any documentation.
I can't create a DataFrame with the info all requests together.
I have the following route in Django Rest Framework:
from rest_framework.viewsets import ModelViewSet
from rest_framework.renderers import JSONRenderer
from rest_framework.response import Response
class MainViewset(ModelViewSet):
renderer_classes = [JSONRenderer]
authentication_classes = []
permission_classes = []
def alive(self, request):
return Response("API is Alive", 200)
I have a Django test that calls this API route, expecting the JSON string:
def test_base_route(self):
c = Client()
response = c.get('/budget/alive')
self.assertTrue(response.status_code == 200)
self.assertEqual(response.content.decode("UTF-8"), "API is Alive")
However, I get the following error:
def test_base_route(self):
c = Client()
response = c.get('/budget/alive')
self.assertTrue(response.status_code == 200)
> self.assertEqual(response.content.decode("UTF-8"), "API is Alive")
E AssertionError: '"API is Alive"' != 'API is Alive'
E - "API is Alive"
E ? - -
E + API is Alive
I find this strange since I decoded the string. I know it's a simple thing to trim off quotation marks, but what is the right way to serialize a single string as a response and get it back in the content of a response in DRF when sending JSON?
You can use .data for this case:
self.assertEqual(response.data, "API is Alive")
I have an APIVIEW in DRF which I have written some views in it to get some Response, I would like to get a better formatted JSONResponse.
Views.py
class Pay(APIView):
def get(self, request):
url = "https://api.paystack.co/transaction/verify/262762380"
payload = {}
files = {}
headers = {
'Authorization': 'Bearer SECRET_KEY',
'Content-Type': 'application/json'
}
response = requests.request("GET", url, headers=headers, data= payload, files=files)
return Response(response)
This is a pictorial representation of the bad formatted JSONResponse I am getting which I would like to improve. Thanks
So in the case you don't have a Model but rather some data structure you get in some other way (as in a response from a third party API) just return a Response with this data structure as argument.
In this case you are effectively acting as a kind of proxy between the 3rd party API and your client, if that is what you intend (be sure not to leak private data!)
Example:
class Pay(APIView):
def get(self, request):
url = "https://api.paystack.co/transaction/verify/262762380"
payload = {}
files = {}
headers = {
'Authorization': 'Bearer SECRET_KEY',
'Content-Type': 'application/json'
}
# no need for 'data' and 'files' arguments with method GET
response = requests.get(url, headers=headers)
# do some validation, at least check for response.status_code
if response.status_code == 200:
data = response.json()
return Response(data)
There are many more possibilities when using the requests library.
After I get a token from a Post request as shown below:
{ "access_token": "12345", "expires_in": 3600, "token_type": "Bearer" }
I want to use this token in different TestSteps Headers values.
For example I have to make a GET request after I received this token and it have in the header -> Authentification : Bearer + token_value.
So can I write a GroovyScript or something to make this automatically? I'm using ReadyApi.
Regards,
Adrian
Add Script Assertion for the same step where you receive the mentioned response:
Script Assertion this fetches the values from response and creates a project property and set the retrieved value.
//Check if the response is empty or null
assert context.response, "Response is null or empty"
def json = new groovy.json.JsonSlurper().parseText(context.response)
def token = "${json.token_type} ${json.access_token}" as String
log.info "Token will be: ${token}"
//Assing the value at project level property TOKEN
context.testCase.testSuite.project.setPropertyValue('TOKEN', token)
Now the value needs to be set as header to each outgoing request dynamically. i.e., Add Authorization header and its value for the SOAP or REST request type steps. For this, Events feature is going to be used.
Add a SubmitListener.beforeSubmit event and add the below script into it. Please follow the comments inline.
import com.eviware.soapui.impl.wsdl.teststeps.WsdlTestRequestStep
import com.eviware.soapui.impl.wsdl.teststeps.RestTestRequestStep
//Please edit the header name as you wanted
def headerName = 'Authorization'
//a method which sets the headers
def setHttpHeaders(def headers) {
def step = context.getProperty("wsdlRequest").testStep
if (step instanceof RestTestRequestStep || step instanceof WsdlTestRequestStep) {
def currentRequest = step.httpRequest
def existingHeaders = currentRequest.requestHeaders
headers.each {
existingHeaders[it.key] = it.value
}
currentRequest.requestHeaders = existingHeaders
} else {
log.info 'not adding headers to the current step as it is not request type step'
}
}
//read the token from project properties
def token = context.expand('${#Project#TOKEN}')
//assert the value of token
assert token, "Token is null or empty"
//UPDATE from the comment to add the header to next request
if (token) {
def headerValue = [(token)]
def headers = [(headerName) : (headerValue)]
setHttpHeaders(headers)
}
import groovy.json.JsonSlurper
import groovy.json.*
def tokens=testRunner.runStepByname("Token")
def response = context.expand( '${Token#Response}' )
def JsonSlurperjsonSlurper = newJsonSlurper()
def Objectresult = jsonSlurper.parseText(response)
def access_token= result.access_token
def authorization = "Bearer "+access_token
testRunner.testCase.setPropertyValue("access_token", authorization)