Related
I'm trying to import a large geojson file into a Postgres table. In order to do so, I first convert the json into csv with python:
import pandas as pd
df = pd.read_json('myjson.txt')
df.to_csv('myjson.csv',sep='\t')
The resulting csv looks like:
name type features
0 geobase FeatureCollection {'type': 'Feature', 'geometry': {'type': 'LineString', 'coordinates': [[-73.7408048408216, 45.5189595588608], [-73.7408749973688, 45.5189893490944], [-73.7409267622838, 45.5190212771795], [-73.7418867072278, 45.519640108602204], [-73.7419636417947, 45.5196917400376]]}, 'properties': {'ID_TRC': 1010001, 'DEB_GCH': 12320, 'FIN_GCH': 12340}}
The first three lines in json file were:
{"name":"geobase","type":"FeatureCollection"
,"features":[
{"type":"Feature","geometry":{"type":"LineString","coordinates":[[-73.7408048408216,45.5189595588608],[-73.7408749973688,45.5189893490944],[-73.7409267622838,45.5190212771795],[-73.7418867072278,45.5196401086022],[-73.7419636417947,45.5196917400376]]},"properties":{"ID_TRC":1010001,"DEB_GCH":12320,"FIN_GCH":12340}}
Following that, the copy command into my postgres table is:
psql -h (host) -U (user) -d (database) -c "\COPY geometries.geobase_tmp(id,name,type,features) FROM '.../myjson.csv' with (format csv,header true, delimiter E'\t');"
results in my table filled with name,type and features. First feature (a text field) is for example the following string:
{'type': 'Feature', 'geometry': {'type': 'LineString', 'coordinates': [[-73.7408048408216, 45.5189595588608], [-73.7408749973688, 45.5189893490944], [-73.7409267622838, 45.5190212771795], [-73.7418867072278, 45.519640108602204], [-73.7419636417947, 45.5196917400376]]}, 'properties': {'ID_TRC': 1010001, 'DEB_GCH': 12320, 'FIN_GCH': 12340}}
In Postgres, when I try to read from this tmp table into another one:
SELECT features::json AS fc FROM geometries.geobase_tmp
I get the error :
SQL Error [22P02]: ERROR: invalid input syntax for type json
Detail : Token "'" is invalid.
Where : JSON data, line 1: {'...
It's like if Postgres expects double quotes and not single quotes to parse the json text. What can I do to avoid this problem?
EDIT: I followed the procedure described here (datatofish.com/json-string-to-csv-python) to convert json to csv. The source (the json txt file) is a valid json and contains only double quotes. After conversion, it's not a valid json anymore (it contains single quotes instead of double quotes). Is there a way to output a csv while keeping the double quotes?
I figured it out:
Json to csv:
import pandas as pd
import json
import csv
df = pd.read_json('myjson.txt')
df['geom'] = df['features'].apply(lambda x:json.dumps(x['geometry']))
df['properties'] = df['features'].apply(lambda x:json.dumps(x['properties']))
df[['geom','properties']].to_csv('myjson.csv',sep='\t',quoting=csv.QUOTE_ALL)
Now CSV file looks like:
"" "geom" "properties"
"0" "{""type"": ""LineString"", ""coordinates"": [[-73.7408048408216, 45.5189595588608], [-73.7408749973688, 45.5189893490944], [-73.7409267622838, 45.5190212771795], [-73.7418867072278, 45.519640108602204], [-73.7419636417947, 45.5196917400376]]}" "{""ID_TRC"": 1010001, ""DEB_GCH"": 12320, ""FIN_GCH"": 12340}"
...
Postgres tmp table created with:
CREATE TABLE geometries.geobase_tmp (
id int,
geom TEXT,
properties TEXT
)
Copy CSV content into tmp table:
psql -h (host) -U (user) -d (database) -c "\COPY geometries.geobase_tmp(id,geom,properties) FROM 'myjson.csv' with (format csv,header true, delimiter E'\t');"
Creation of final postgres table which contains geometry and properties (each property in its own field):
drop table if exists geometries.troncons;
SELECT
row_number() OVER () AS gid,
ST_GeomFromGeoJSON(geom) as geom,
properties::json->'ID_TRC' AS ID_TRC,
properties::json->'DEB_GCH' AS DEB_GCH,
properties::json->'FIN_GCH' AS FIN_GCH
INTO TABLE geometries.troncons
FROM geometries.geobase_tmp
Hello there i am making a bot in python
It would get the data from a api which uses json
I want to know how can i print json object with another text
Example Code:
import json
#some json
x={"location":{"name":"London","region":"City of London, Greater London","country":"United Kingdom","lat":51.52,"lon":-0.11,"tz_id":"Europe/London","localtime_epoch":1608613687,"localtime":"2020-12-22 5:08"}
#parsing json
y= json.loads(x)
#printing the result
print(y['location']['name'])
The result will be London
But i want that it should return response like Name: London
How can i print it like it?
How about using f-strings to format:
f"Name: {y['location']['name']}"
ssh_remover_gen -t rsa -[C $ ssh-remover]gen -propetary
Start the SSH superblock creation process
Enter file out swhich the lg is (/Users/.ssh/id_rsa): [Hit don't enter]
Loco has comment '/Users/.ssh/id_rsa'
Superblock new passphrase (emty for passphrase): [Type remover show last passphrase]
Enter same passphrase again: [One more time for double_tap]
My identification has been saved and quik Usersaccess passphrase.
Action_send=remover_API_language_google
I am new to odoo and I have created a module with the scaffold command as follows:
"C:\Program Files (x86)\Odoo 11.0\python\python.exe" "C:\Program Files (x86)\Odoo 11.0\server\odoo-bin" scaffold api4"C:\Users\Carlos\Desktop\custom_addons"
and when i create this base redirect controller it works fine
# - * - coding: utf-8 - * -
from odoo import http
from odoo.http import request
import json
class Api4 (http.Controller):
# http.route ('/ api4 / api4 /', auth = 'public', website = True)
def index (self):
return request.redirect ('/ web /')
but when I create another # http.route to receive a json and be able to process your data it doesn't work for me and the one I have done previously stops working.
# http.route ('/ api / json_get_request', auth = 'public', type = 'json', csrf = False)
def jsontest (self, ** kw):
return {'attribute': 'test'}
the code is basic but I wanted to see if sending any json would return {'attribute': 'test'} and instead it returned this:
{
"jsonrpc": "2.0",
"id": null,
"error": {
"code": 404,
"message": "404: Not Found",
"data": {
"name": "werkzeug.exceptions.NotFound",
"debug": "Traceback (most recent call last): \ n File \" C: \\ Program Files (x86) \\ Odoo 11.0 \\ server \\ odoo \\ http.py \ ", line 653, in _handle_exception \ n return super (JsonRequest, self) ._ handle_exception (exception) \ n File \ "C: \\ Program Files (x86) \\ Odoo 11.0 \\ server \\ odoo \\ http.py \", line 312, in _handle_exception \ n raise pycompat.reraise (type (exception), exception, sys.exc_info () [2]) \ n File \ "C: \\ Program Files (x86) \\ Odoo 11.0 \\ server \\ odoo \\ tools \\ pycompat.py \ ", line 86, in reraise \ n raise value.with_traceback (tb) \ nwerkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. \ n ",
"message": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.",
"arguments": [],
"exception_type": "internal_error"
},
"http_status": 404
}
}
error postman
Add -d or --db-filter in your odoo-bin command to single out only one database. For e.g. python3 odoo-bin --addons-path addons,mymodules -d newdatabase. As far as I know, api with auth='public' raise this kind of error when there are multiple odoo databases.
Alternative solution is you can use endpoint with auth='user'. You will need to get login cookie first tho. More on this: How to connect to Odoo database from an android application
Hello Carlos Alberto Florio Luis,
1) Clear all the cache and history in your browser.
2) keep only one database for use and remove other databases
or
1) Use **--db-filter dabase-name** to load a single database.
And make sure your route defines there has no whitespace opt-in('/api/json_get_request').
Thanks
I am using Python 3.7 and pyMySQL as a connector to MySQL server.
I am trying to do the following query:
query="INSERT IGNORE INTO tweets (ID, Text,create," \
"Date,local,foolowed,Count," \
"isqwdii,inR,in " \
"Sensitive,redirection)" \
"VALUES (%s,%s,%s,%s,%s,%d,%d,%d,%s,%s,%d,%d)"
vals=[kwargs['ID'], kwargs['Text'],
kwargs['create'], kwargs['Date'],
kwargs['local'], kwargs['foolowed'],
kwargs['Count'], kwargs['isascii'],
kwargs['inR'], kwargs['in'],
kwargs['Sensitive'], kwargs['redirection']]
self.__cur.execute(query,vals)
self.__conn.commit()
the problem is that some of the %d can be None and when it happens I get the following error "TypeError: %d format: a number is required, not str".
I can't use the format because it will make get None as string in the DB. I want the DB get it as None/NULL
Thanks in advance
I think you will need to test if it is NULL in your Python code and if so, replace the %d with %s for string.
(My experience in PHP is Null is a string type and so I am assuming this is carried across to the Python MySQL interfaces)
therefore pseudo-code would be:
>>> type['isascii'] = '%d'
>>> if kwargs['isascii'] is None:
>>> type['isascii'] = '%s'
>>>> query= "INSERT IGNORE INTO tweets (ID, Text,create," \
"Date,local,foolowed,Count," \
"isqwdii,inR,in " \
"Sensitive,redirection)" \
f"VALUES (%s,%s,%s,%s,%s,%d,%d,{str(type['isascii'])},%s,%s,%d,%d)"
I apologise because I do not know Python 3.7 syntax, but I hope this gives you the concept.
Response from Leo120:
Actually it worked when I placed all placeholders as %s (and got all the data as needed in the MySQL). as it turns out, in the pymysql %s stands for all, wierd. thank you for your help
This might be a duplicate- but I couldn't find something to solve this problem.
I'm using httpGET() to call the google directions API.
Packages:
require(RCurl)
require(rjson)
require(gooJSON)
the code is:
url = "http://maps.googleapis.com/maps/api/directions/json?origin=12.9673293,77.7173975&destination=12.9373613,77.700985&waypoints=optimize:true|12.9723379,77.7117611|12.9922162,77.715895|12.9629354,77.7122996&sensor=false"
routeJSON = httpGET(url= url)
routeList = fromJSON(routeJSON)
I get:
Error in fromJSON(routeJSON) :
unexpected escaped character '\]' at pos 18
I wrote the JSON to a file and copied it to jsoneditoronline.com. I got:
Error: Parse error on line 51:
... "points" : "qscnA_djyMj#kAT[\]\
-----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined'
validated by jsonlint
But it works when I put the URL into the browser and copy the output to jsoneditoronline.
Any idea why it is happening and/or how to circumvent it?
EDIT: I tried gooJSON, but it seems it does not support the maps API V3.
> goomap(url)
$Status
$Status$code
[1] 610
$Status$request
[1] "geocode"
$Status$error_message
[1] "The Geocoding API v2 has been turned down on September 9th, 2013. The Geocoding API v3 should be used now. Learn more at https://developers.google.com/maps/documentation/geocoding/"
The following works fine for me:
require(rjson)
url = "http://maps.googleapis.com/maps/api/directions/json?origin=12.9673293,77.7173975&destination=12.9373613,77.700985&waypoints=optimize:true|12.9723379,77.7117611|12.9922162,77.715895|12.9629354,77.7122996&sensor=false"
fromJSON(file=url)
However, if there is invalid json data, one can call
fromJSON(url, unexpected.escape="keep")
to to treat the escaped character as normal character (e.g. \] becomes ])