Should only be done with python3.
As i am not familiar with python3 i would like a full answer
and not just a snippet. Thank you if you take this into consideration!
curl -s "http://kodi:kodi#192.168.1.10:8080/jsonrpc?Base" -H 'Content-Type: application/json' --data '[{"jsonrpc":"2.0","method":"Player.GetProperties","params":[1,["time"]],"id":17},{"jsonrpc":"2.0","method":"Player.GetItem","params":[1,["file"]],"id":18}]'
this is what I get for the curl command:
[{"id":17,"jsonrpc":"2.0","result":{"time":{"hours":0,"milliseconds":597,"minutes":44,"seconds":53}}},{"id":18,"jsonrpc":"2.0","result":{"item":{"file":"smb://192.168.1.10/#.mkv","id":340,"label":"The Expendables 3","type":"movie"}}}]
from which I would get the values of "file", "hours", "minutes", and "seconds"
in front of hours, minutes, and seconds, if the number is 0-9 then put a 0 in front of it and save it in a .txt file
the .txt file should look like this:
smb://192.168.1.10/#.mkv
004453
exaple 2.
smb://192.168.1.10/#.mkv
010703
example 3.
smb://192.168.1.10/#.mkv
025547
bash solution, and more information here:
I got very good answers to my last question about this in bash
(if it might help you, here are complete solutions in awk or jq)
but, now i have to move to python3..
Thanks in advance for any help!
Edit3:
I finally succeeded in extracting the values.
how can this be further filtering?
to get it as shown in the example below?
>>> import json
>>> import base64
>>> from urllib.request import Request, urlopen
>>>
>>> credentials = b'kodi:kodi'
>>> encoded_credentials = base64.b64encode(credentials)
>>> authorization = b'Basic ' + encoded_credentials
>>>
>>> headers = { 'Content-Type': 'application/json', 'Authorization': authorization }
>>> url = 'http://192.168.1.10:8080/jsonrpc'
>>> data = [{"jsonrpc":"2.0","method":"Player.GetProperties","params":[1,["time"]],"id":17},{"jsonrpc":"2.0","method":"Player.GetItem","params":[1,["file"]],"id":18}]
>>>
>>> json_data = json.dumps(data)
>>> post_data = json_data.encode('utf-8')
>>> request = Request(url, post_data, headers)
>>> result = urlopen(request)
>>> print(result.read())
b'[{"id":17,"jsonrpc":"2.0","result":{"time":{"hours":0,"milliseconds":914,"minutes":52,"seconds":59}}},{"id":18,"jsonrpc":"2.0","result":{"item":{"file":"smb://192.168.1.10/n filmek 720p/The.Expendables.3.2014.Hybrid.READ.NFO.Extended.720p.BluRay.DTS.x264.HuN-TRiNiTY/theexpendables3.720p.hybrid.ex-trinity.mkv","id":340,"label":"The Expendables 3","type":"movie"}}}]'
which I would like to filter further for this:
smb://192.168.1.10/#.mkv
0
52
59
and I would like to "convert" it to this
smb://192.168.1.10/#.mkv
005259
import json
import base64
import requests
credentials = b'kodi:kodi'
encoded_credentials = base64.b64encode(credentials)
authorization = b'Basic ' + encoded_credentials
headers = { 'Content-Type': 'application/json', 'Authorization': authorization }
url = 'http://192.168.1.10:8080/jsonrpc?Base'
data = [
{"jsonrpc":"2.0","method":"Player.GetProperties","params":[1,["time"]],"id":17},
{"jsonrpc":"2.0","method":"Player.GetItem","params":[1,["file"]],"id":18}]
response = requests.post(url, headers=headers, data=json.dumps(data))
result = response.json()
h,m,s = map(int, [result[0]['result']['time'][val] for val in ['hours','minutes','seconds']])
time = f'{h:02}{m:02}{s:02}'
print(result[1]['result']['item']['file'],time, sep='\n', file=open("c:\\kodi\\save.txt", "w"))
save.txt
smb://192.168.1.10/#.mkv
005259
Related
I can query all occurances of certain base url within a given common crawl index, saving them all to a file and get a specific article (test_article_num) using the code below. However, I have not come across a way to extract the raw html for that article from the specific crawl-data ('filename' in output), even though I know the offset and length of the data I want. I feel like there should be a way to do this in python similar to this, maybe using requests and warcio (perhaps something akin to this), but I'm not sure. Any help is greatly appreicated.
EDIT:
I found exactly what I needed here.
import requests
import pathlib
import json
news_website_base = 'hobbsnews.com'
URL = "https://index.commoncrawl.org/CC-MAIN-2022-05-index?url="+news_website_base+"/*&output=json"
website_output = requests.get(URL)
pathlib.Path('data.json').write_bytes(website_output.content)
news_articles = []
test_article_num=300
for line in open('data.json', 'r'):
news_articles.append(json.loads(line))
print(news_articles[test_article_num])
news_URL=news_articles[test_article_num]['url']
news_warc_file=news_articles[test_article_num]['filename']
news_offset=news_articles[test_article_num]['offset']
news_length=news_articles[test_article_num]['length']
Code output:
{'urlkey': 'com,hobbsnews)/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/{{%20data.link', 'timestamp': '20220122015439', 'url': 'https://www.hobbsnews.com/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/%7B%7B%20data.link', 'mime': 'text/html', 'mime-detected': 'text/html', 'status': '404', 'digest': 'GY2UDG4G3V3S5TXDL3H7HE6VCSRBD3XR', 'length': '40062', 'offset': '21016412', 'filename': 'crawl-data/CC-MAIN-2022-05/segments/1642320303729.69/crawldiagnostics/CC-MAIN-20220122012907-20220122042907-00614.warc.gz'}
https://www.hobbsnews.com/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/%7B%7B%20data.link
crawl-data/CC-MAIN-2022-05/segments/1642320300343.4/crawldiagnostics/CC-MAIN-20220117061125-20220117091125-00631.warc.gz
21016412
40062
With the WARC URL, and WARC record offset and length it's simply:
download the range from offset until offset+length-1
pass the downloaded bytes to a WARC parser
Using curl and warcio CLI:
curl -s -r250975924-$((250975924+6922-1)) \
https://data.commoncrawl.org/crawl-data/CC-MAIN-2021-10/segments/1614178365186.46/warc/CC-MAIN-20210303012222-20210303042222-00595.warc.gz \
>warc_temp.warc.gz
warcio extract --payload warc_temp.warc.gz 0
Or with Python requests and warcio (cf. here):
import io
import requests
import warcio
warc_filename = 'crawl-data/CC-MAIN-2021-10/segments/1614178365186.46/warc/CC-MAIN-20210303012222-20210303042222-00595.warc.gz'
warc_record_offset = 250975924
warc_record_length = 6922
response = requests.get(f'https://data.commoncrawl.org/{warc_filename}',
headers={'Range': f'bytes={warc_record_offset}-{warc_record_offset + warc_record_length - 1}'})
with io.BytesIO(response.content) as stream:
for record in warcio.ArchiveIterator(stream):
html = record.content_stream().read()
I'm sending following REST request to the Read The Docs API:
GET /api/v3/projects/<my_project>/redirects/
I receive JSON in the response body, but I'm unable to parse the JSON into something useful.
I'm using a Python3 program:
import requests
import json
URL = 'https://readthedocs.com/api/v3/projects/<my_project>/redirects/'
TOKEN = "<my_access_token>"
HEADERS = {'Authorization': f'token {TOKEN}'}
response = requests.get(URL, headers=HEADERS, stream=True)
print(f'Response is {response.json()}')
This program returns something that looks like JSON.
If I run it using the following command line:
python3 rtd_get_redirects.py | python3 -m json.tool
I receive the following error:
Expecting value: line 1 column 1 (char 0)
The entire response is large, so I'll just paste this snippet from the beginning:
{'count': 521, 'next': 'https://readthedocs.com/api/v3/projects/signalfx-product-docs/redirects/?limit=10&offset=10', 'previous': None, 'results': [{'_links': {'_self': 'https://readthedocs.com/api/v3/projects/signalfx-product-docs/redirects/6612/', 'project': 'https://readthedocs.com/api/v3/projects/signalfx-product-docs/'}, 'created': '2022-07-20T18:08:52.133088Z',
i have a server with POST e REST services. Using micropython i need to perform POST request.
GET queries arrive at the REST services and respond correctly using the following format:
s.send(b"GET /sensordata/premises HTTP/1.1\r\nHost:XX.XXX.XXX.XXX\r\n" + "Accept: application/json\r\n\r\n")
but for the POST looks like the request arrived to server, but the body is empty. For some reason the JSON body of the request is not interpreted in a correct way.
from network import WLAN
import socket
import machine
import time
import struct
import json
import pycom
wlan = WLAN(mode=WLAN.STA)
wlan.connect("*****", auth=(WLAN.WPA2, "*****"), timeout=5000)
while not wlan.isconnected():
machine.idle()
print("Connected to WiFi\n")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
url = 'XX.XXX.XXX.XXX'
sockaddr = socket.getaddrinfo(url, 80) [0][-1]
s.connect(sockaddr)
print('socket connected')
httpreq = b'POST /sensordata/insertrecords HTTP/1.1 \r\n Host:XX.XXX.XXX.XXX \r\n' + 'Accept: application/json \r\n' + 'Content-Type: application/json\r\n' + ' {\"node_id\":\"1\",\"value\":[{\"measure_time_stamp\":\"2020-10-06T09:25:43\",\"temp\":\"14\",\"humidity\":\"75\",\"ph1\":11,\"ph2\":12,\"ph3\":13}]}\r\n\r\n'
s.send(httpreq)
time.sleep(1)
rec_bytes = s.recv(4096)
print("RESPONSE = " + str(rec_bytes))
print('end')
s.close()
Are you aware of the urequests library? It's an HTTP client library inspired by the popular Python requests. A POST request is simply:
import urequests
r = urequests.post('https://your/endpoint')
I'm trying to read some json file from s3 bucket and then trying to post the data into rds and redshift(using a post api developed/intended for that).
I approached to do this in 2 ways using boto3. Below are those:
1st way:
import boto3
import json
import requests
url = 'xxxx.us-east-1.elb.amazonaws.com/v1'
headers = {'content-type': 'application/json', 'Host': 'development.my.dns.com'}
endpoint = url+'/my/post/endpoint'
s3_client = boto3.client("s3")
fileObj = s3_client.get_object(Bucket='my-bucket-name', Key='my-key-name'])
data = fileObj['Body'].read().decode('utf-8')
with requests.request('POST', endpoint, data=data, headers=headers, auth=(username, pwd), verify=False, stream=True) as r:
print("Status Code:",r.status_code)
2nd way:
import boto3
import json
import requests
url = 'xxxx.us-east-1.elb.amazonaws.com/v1'
headers = {'content-type': 'application/json', 'Host': 'development.my.dns.com'}
endpoint = url+'/my/post/endpoint'
s3_res = boto3.resource('s3')
contentObj = s3_res.Object('my-bucket-name', 'my-key-name')
fileContent = contentObj.get()['Body'].read().decode('utf-8')
data = json.dumps(json.loads(fileContent))
with requests.request('POST', endpoint, data=data, headers=headers, auth=(username, pwd), verify=False, stream=True) as r:
print("Status Code:",r.status_code)
Basically everything is same(url, endpoint, headers, requests.request) and type(data) is <class 'str'> in both the approaches. But the status codes are always different.
The 2nd way gives Status Code: 200. But the 1st way gives Status Code: 413 or Status Code: 502 randomly
Can any one please explain why is it this way? what's different in the above two approaches? I'm trying to understand what's going on when boto3 read() the data differently.
I just began exploring APIs. This is my code so far. For locu API this works but for Zomato they use curl header request which I don't know how to use. Could someone guide or show me how?
import json
import urllib2
Key = 'Zomato_key'
url = 'https://developers.zomato.com/api/v2.1/categories'
json_obj = urllib2.urlopen(url)
data = json.load(json_obj)
print data
By looking at the Zomato API docs, it seems that the parameter user-key has to be set in the header.
The following works:
import json
import urllib2
Key = '<YOUR_ZOMATO_API_KEY>'
url = "https://developers.zomato.com/api/v2.1/categories"
request = urllib2.Request(url, headers={"user-key" : Key})
json_obj = urllib2.urlopen(request)
data = json.load(json_obj)
print data
If you want a more elegant way to query APIs, have a look at requests module (you can install using pip install requests).
I suggest you the following:
import json
import requests
Key = <YOUR_ZOMATO_API_KEY>'
url = "https://developers.zomato.com/api/v2.1/categories"
if __name__ == '__main__':
r = requests.get(url, headers={'user-key': Key})
if r.ok:
data = r.json()
print data
NB: I suggest you remove your Key from StackOverflow if you care about keeping it to yourself.
this didn't work for me can u suggest some other method for me.
-->the code when tried to compile is taking long time and returning an traceback error in request method that is in built
but curl command is working
curl -X GET --header "Accept: application/json" --header "user-key: c5062d18e16b9bb9d857391bb32bb52f" "https://developers.zomato.com/api/v2.1/categories"