I'm trying to query a PostGIS table using GeoAlchemy to return the row where the point lies inside. I've a point and I'm using the timezones shapefile from natural earth. I've tried the below but it just returns the query and not the row item.
class TimeZonePoly(Base):
__tablename__ = 'time_zone_poly_grid_exp'
__table_args__ = {'autoload': True}
def PointInside(PostGIS_Table, Lat, Lon):
point = func.ST_GeographyFromText('POINT({} {})'.format(Lon, Lat))
return session.query(PostGIS_Table).filter(func.ST_Contains(PostGIS_Table.geom, point))
row = PointInside(TimeZonePoly, 47, -2)
print(row)
which returns the follow:
SELECT time_zone_poly_grid_exp.id AS time_zone_poly_grid_exp_id, ST_AsEWKB(time_zone_poly_grid_exp.geom) AS time_zone_poly_grid_exp_geom, time_zone_poly_grid_exp.objectid AS time_zone_poly_grid_exp_objectid, time_zone_poly_grid_exp.scalerank AS time_zone_poly_grid_exp_scalerank, time_zone_poly_grid_exp.featurecla AS time_zone_poly_grid_exp_featurecla, time_zone_poly_grid_exp.name AS time_zone_poly_grid_exp_name, time_zone_poly_grid_exp.map_color6 AS time_zone_poly_grid_exp_map_color6, time_zone_poly_grid_exp.map_color8 AS time_zone_poly_grid_exp_map_color8, time_zone_poly_grid_exp.note AS time_zone_poly_grid_exp_note, time_zone_poly_grid_exp.zone AS time_zone_poly_grid_exp_zone, time_zone_poly_grid_exp.utc_format AS time_zone_poly_grid_exp_utc_format, time_zone_poly_grid_exp.time_zone AS time_zone_poly_grid_exp_time_zone, time_zone_poly_grid_exp.iso_8601 AS time_zone_poly_grid_exp_iso_8601, time_zone_poly_grid_exp.places AS time_zone_poly_grid_exp_places, time_zone_poly_grid_exp.dst_places AS time_zone_poly_grid_exp_dst_places, time_zone_poly_grid_exp.tz_name1st AS time_zone_poly_grid_exp_tz_name1st, time_zone_poly_grid_exp.tz_namesum AS time_zone_poly_grid_exp_tz_namesum
FROM time_zone_poly_grid_exp
WHERE ST_Contains(time_zone_poly_grid_exp.geom, ST_GeographyFromText(%(ST_GeographyFromText_1)s))
It looks like it's the correct statement, I just can't get the results!
Related
I have a DB collection consisting of nested strings . I am trying to convert the contents under "status" column as separate columns against each order ID in order to track the time taken from "order confirmed" to "pick up confirmed". The string looks as follows:
I have tried the same using
xyz_db= db.logisticsOrders -------------------------(DB collection)
df =pd.DataFrame(list(xyz_db.find())) ------------(JSON to dataframe)
Using normalize :
parse1=pd.json_normalize(df['status'])
It works fine in case of non nested arrays. But status being a nested array the output is as follows:
Using for :
data = df[['orderid','status']]
data = list(data['status'])
dfy = pd.DataFrame(columns = ['statuscode','statusname','laststatusupdatedon'])
for i in range(0, len(data)):
result = data[i]
dfy.loc[i] = [data[i][0],data[i][0],data[i][0],data[i][0]]
It gives the result in form of appended rows which is not the format i am trying to achieve
The output I am trying to get is :
Please help out!!
i share you which i used json read, maybe help you:
you can use two and more list
def jsonify(z):
genr = []
if z==z and z is not None:
z = eval(z)
if type(z) in (dict, list, tuple):
for dic in z:
for key, val in dic.items():
if key == "name":
genr.append(val)
else:
return None
else:
return None
return genr
top_genr['genres_N']=top_genr['genres'].apply(jsonify)
I'm trying to create a GeoDataFrame with 2 zip codes per row, whose distances from each other I want to compare.
I took a list of approx 220 zip codes and ran an itertools combination on them to get all combo's, then unpacked the tuples into two columns
code_combo = list(itertools.combinations(df_with_all_zip_codes['code'], 2))
df_distance_ctr = pd.DataFrame(code_combo, columns=['first_code','second_code'])
Then I did some standard pandas merges and column renaming to get the polygon/geometry column from the original geodataframe into this new one, right beside the respective zip code columns.
The problem is I can't seem to get the polygon columns to be read as geometry, even after 1.) attempting to convert the dataframe to a geodataframe - AttributeError: No geometry data set yet, 2.) applying wkt.loads to the geometry column - AttributeError: 'MultiPolygon' object has no attribute 'encode'
.
I've tried to look for a way to convert a series to a geoseries but can't find anything on SO nor the documentation. Can anyone please point out where I'm likely going wrong?
Looking at the __init__ method of a GeoDataFrame at https://github.com/geopandas/geopandas/blob/master/geopandas/geodataframe.py, it looks like a GDF can only have one column at a time. The other columns you've created should still have geometry objects in them though.
Since you still have geometry objects in each column, you could write a method that uses Shapely's distance method, like so:
import pandas as pd
import geopandas
from shapely.geometry import Point
import matplotlib.pyplot as plt
lats = [-34.58, -15.78, -33.45, 4.60, 10.48]
lons = [-58.66, -47.91, -70.66, -74.08, -66.86]
df = pd.DataFrame(
{'City': ['Buenos Aires', 'Brasilia', 'Santiago', 'Bogota', 'Caracas'],
'Country': ['Argentina', 'Brazil', 'Chile', 'Colombia', 'Venezuela'],
'Latitude': lats,
'Longitude': lons})
df['Coordinates'] = list(zip(df.Longitude, df.Latitude))
df['Coordinates'] = df['Coordinates'].apply(Point)
df['Coordinates_2'] = list(zip(lons[::-1], lats[::-1]))
df['Coordinates_2'] = df['Coordinates_2'].apply(Point)
gdf = geopandas.GeoDataFrame(df, geometry='Coordinates')
def get_distance(row):
distance = row.Coordinates.distance(row.Coordinates_2)
print(distance)
return distance
gdf['distance'] = gdf.apply(lambda row: get_distance(row), axis=1)
As for the AttributeError: 'MultiPolygon' object has no attribute 'encode'. MultiPolygon is a Shapely geometry class. encode is usually a method on string objects so you can probably remove the call to wkt.loads.
I am having a problem with web-scraping. I am trying to learn how to do it, but I can't seem to get past some of the basics. I am getting an error, "TypeError: 'ResultSet' object is not callable" is the error I'm getting.
I've tried a number of different things. I was originally trying to use the "find" instead of "find_all" function, but I was having an issue with beautifulsoup pulling in a nonetype. I was unable to create an if loop that could overcome that exception, so I tried using the "find_all" instead.
page = requests.get('https://topworkplaces.com/publication/ocregister/')
soup = BeautifulSoup(page.text,'html.parser')all_company_list =
soup.find_all(class_='sortable-table')
#all_company_list = soup.find(class_='sortable-table')
company_name_list_items = all_company_list('td')
for company_name in company_name_list_items:
#print(company_name.prettify())
companies = company_name.content[0]
I'd like this to pull in all the companies in Orange County California that are on this list in a clean manner. As you can see, I've already accomplished pulling them in, but I want the list to be clean.
You've got the right idea. I think instead of immediately finding all the <td> tags (which is going to return one <td> for each row (140 rows) and each column in the row (4 columns)), if you want only the company names, it might be easier to find all the rows (<tr> tags) then append however many columns you want by iterating the <td>s in each row.
This will get the first column, the company names:
import requests
from bs4 import BeautifulSoup
page = requests.get('https://topworkplaces.com/publication/ocregister/')
soup = BeautifulSoup(page.text,'html.parser')
all_company_list = soup.find_all('tr')
company_list = [c.find('td').text for c in all_company_list[1::]]
Now company_list contains all 140 company names:
>>> print(len(company_list))
['Advanced Behavioral Health', 'Advanced Management Company & R³ Construction Services, Inc.',
...
, 'Wes-Tec, Inc', 'Western Resources Title Company', 'Wunderman', 'Ytel, Inc.', 'Zillow Group']
Change c.find('td') to c.find_all('td') and iterate that list to get all the columns for each company.
Pandas:
Pandas is often useful here. The page uses multiple sorts including company size, rank. I show rank sort.
import pandas as pd
table = pd.read_html('https://topworkplaces.com/publication/ocregister/')[0]
table.columns = table.iloc[0]
table = table[1:]
table.Rank = pd.to_numeric(table.Rank)
rank_sort_table = table.sort_values(by='Rank', axis=0, ascending = True)
rank_sort_table.reset_index(inplace=True, drop=True)
rank_sort_table.columns.names = ['Index']
print(rank_sort_table)
Depending on your sort, companies in order:
print(rank_sort_table.Company)
Requests:
Incidentally, you can use nth-of-type to select just first column (company names) and use id, rather than class name, to identify the table as faster
import requests
from bs4 import BeautifulSoup as bs
r = requests.get('https://topworkplaces.com/publication/ocregister/')
soup = bs(r.content, 'lxml')
names = [item.text for item in soup.select('#twpRegionalList td:nth-of-type(1)')]
print(names)
Note the default sorting is alphabetical on name column rather than rank.
Reference:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_values.html
I am using odoo 10 and I have two models Order_Line and Products.
OrderLine
class OrderLine(models.Model):
_name = 'order_line'
_description = 'Order Lines'
name = fields.Char()
products = fields.Many2one('amgl.products', String='Products')
Products
class Products(models.Model):
_name = 'products'
_description = 'Products'
_sql_constraints = [
('uniq_poduct_code', 'unique(product_code)', 'Product Code already exists!')
]
name = fields.Char()
product_code = Char()
Now i am trying to create order_line from a csv file and in csv file the customer is providing me 'Product Code' instead of Id. How to handle this that, we use product code and system automatically fills the products associated with that product code.
Note :
Product Code in products table is also unique, so there is no chance of duplicating.
CSV template:
customer/account_number,customer/first_name,customer/last_name,customer/account_type,order/transaction_id,order/products/product_code,order/quantity,order/customer_id/id
Case 1: there are no products stored in the database with any of the product codes the customer is giving to you
If the product codes haven't been created yet in the database, you should have two CSV files (Products.csv and OrderLine.csv). The first one must have three columns (id, name and product_code). The second one must have three columns too (id, name and products/id). So you would only have to make up a XML ID under the id column in Products.csv and call this XML ID from the respective row of the column products/id of the file OrderLine.csv.
Case 2: the product codes the customer has given to you belong to existing products in the database
Now, the customer has given you product codes of products which already exist in the database. In this case you don't have to create a Products.csv file. You need to know which are the XML IDs of the products which have the product codes the customer gave to you. For that, you can go through the interface of Odoo to the tree view of the model products (if this view doesn't exist, you must create it). Then, you'll have to select all records (click on the number 80 of the top right corner to show more records per page if you need it). Once all of them are selected, click on More button and afterwars on Export. Select the column product_code and name and afterwards proceed. Save the generated CSV file as Products.csv, for example. Open it, you'll see all the XML ID of the exported products (if they hadn't XML ID, after the exportation they'll do -an exportation generates XML ID for each exported record if it doesn't have anyone-). Now, I guess the customer has given you something like a file with columns Name of the order line, Product code, so replace the Product code column values with the respective XML IDs of the products you have just exported. So in the end youu should have one file to import, OrderLine.csv, with id, name and products/id columns.
Case 3: there are some product codes belonging to existing products stored in the database and there are some ones which still don't exist
In this case you will have to combine both cases 1 and 2, first, export the products as described in case 2, and then, create a new one with the products whose code doesn't exist yet, as described in case 1. Then replace the product codes the customer gave to you with the respective ones as described in case 2.
Note: this process will give you a lot of time if you have thousands of records to import and you replace them manually. In this case it is mandatory to create a macro in your CSV editor which does the replacements (with search and replace). For example, with LibreOffice you can do macros with Python.
Example (Case 3)
The customer has given you a file of order lines, with two lines:
Name: OL A, Product Code: AAA
Name: OL B, Product Code: BBB
You export products from Odoo interface and you get a file with one
line:
id,name,product_code
__export__.products_a,"Product A","AAA"
You look for the coincidences of the product codes in both files, and
do the replacements in a copy of the customer file, so now you have
this:
Name: OL A, Product Code: __export__.products_a
Name: OL B, Product Code: BBB
Then you create a new CSV Products.csv and put in there the products
whose product code don't exist yet:
id,name,product_code
__import__.products_b,"Product B","BBB"
Now apply the replacements again comparing this new file with the one
we had, and you will get this:
Name: OL A, Product Code: __export__.products_a
Name: OL B, Product Code: __import__.products_b
Convert this file to a right CSV format for Odoo, and save it as
OrderLine.csv:
id,name,products/id
__import__.order_line_1,"OL A",__export__.products_a
__import__.order_line_2,"OL B",__import__.products_b
And finally, import the files, and take into account: import
Products.csv before OrderLine.csv.
EDIT
I think it should be better to waste a bit of time in programming a macro for your CSV editor (Excel, LibreOffice, Open Office or whatever), but if you're desperated and you need to do this only through Odoo, I came up with an awful workaround, but at least, it should work too.
1.Create a new Char field named product_code in order_line model (it would be there temporaly).
2.Modify the ORM create method of this model:
#api.model
def create(self, vals):
product_id = False
product_code = vals.get('product_code', False)
if product_code:
product = self.env['products'].search([
('product_code', '=', product_code)
])
if product:
product_id = product[0].id
vals.update({
'products': product_id,
})
return super(OrderLine, self).create(vals)
3.Copy the file which the customer's sent you, rename the headers properly, and rename the column order/products/product_code as product_code. Import the CSV file. Each importation of records will call the ORM create method of order_line model.
After the importation you'll have in the database the order lines rightly related to the products.
When you've finished you'll have to remember to remove the code you've added (and also remove the column product_code from order_line model in the database, in order to remove junk).
Solution 1
You can create a transient model with the fields that you are using in the CSV. And applying the idea of #forvas:
class ImportOrderLines(models.TransientModel):
_name = 'import.order.lines'
product_code = Char()
#api.model
def create(self, vals):
product_id = False
product_code = vals.get('product_code', False)
if product_code:
product = self.env['products'].search([
('product_code', '=', product_code)
])
if product:
product_id = product[0].id
self.env['order_line'].create({
'products': product_id,
})
return False # you don't need to create the record in the transient model
You can go to the list view of this transient model and import like in any other model, with the base_import view.
Solution 2
You could create a wizard in order to import the CSV to create the Order Lines.
Check the following source code. You must assing the method import_order_lines to a button in the wizard.
import base64
import magic
import csv
from cStringIO import StringIO
import codecs
from openerp import models, fields, api, _
from openerp.exceptions import Warning
class ImportDefaultCodeWizard(models.TransientModel):
_name = 'import.default_code.wizard'
name = fields.Char(
string='File name',
)
file = fields.Binary(
string='ZIP file to import to Odoo',
required=True,
)
#api.multi
def import_order_lines(self):
self.ensure_one()
content = base64.decodestring(self.file)
if codecs.BOM_UTF8 == content[:3]: # remove "byte order mark" (windows)
content = content[3:]
file_type = magic.from_buffer(content, mime=True)
if file_type == 'text/plain':
self._generate_order_line_from_csv(content)
return self._show_result_wizard()
raise Warning(
_('WRONG FILETYPE'),
_('You should send a CSV file')
)
def _show_result_wizard(self):
return {
'type': 'ir.actions.act_window',
'res_model': self._name,
'view_type': 'form',
'view_mode': 'form',
'target': 'new',
'context': self.env.context,
}
def _generate_order_line_from_csv(self, data):
try:
reader = csv.DictReader(StringIO(data))
except Exception:
raise Warning(
_('ERROR getting data from csv file'
'\nThere was some error trying to get the data from the csv file.'
'\nMake sure you are using the right format.'))
n = 1
for row in reader:
n += 1
self._validate_data(n, row)
default_code = row.get('default_code', False)
order_line = {
'default_code': self._get_product_id(default_code),
# here you should add all the order line fields
}
try:
self.env['order_line'].create(order_line)
except Exception:
raise Warning(
_('The order line could not be created.'
'\nROW: %s') % n
)
def _validate_data(self, n, row):
csv_fields = [
'default_code',
]
""" here is where you should add the CSV fields in order to validate them
customer/account_number, customer/first_name, customer/last_name,
customer/account_type, order/transaction_id, order/products/product_code ,order/quantity, order/customer_id/id
"""
for key in row:
if key not in csv_fields:
raise Warning(_('ERROR\nThe file format is not right.'
'\nCheck the column names and the CSV format'
'\nKEY: %s' % key))
if row.get('default_code', False) == '':
raise Warning(
_('ERROR Validating data'),
_('The product code should be filled.'
'\nROW: %s') % n
)
def _get_product_id(self, default_code):
if partner_id:
product_obj = self.env['product.product'].search([
('default_code', '=', default_code),
])
if len(product_code_obj) == 1:
return product_obj.default_code
else:
raise Warning(
_('ERROR Validating data'),
_('The product code should be filled.'
'\nROW: %s') % n
)
return False
You can search by product_code like so:
#api.model
def search_by_code(self, code):
result = self.env['products'].search([('product_code', '=', code)])
My goal is to (1) import Twitter JSON, (2) extract data of interest, (3) create pandas data frame for the variables of interest. Here is my code:
import json
import pandas as pd
tweets = []
for line in open('00.json'):
try:
tweet = json.loads(line)
tweets.append(tweet)
except:
continue
# Tweets often have missing data, therefore use -if- when extracting "keys"
tweet = tweets[0]
ids = [tweet['id_str'] for tweet in tweets if 'id_str' in tweet]
text = [tweet['text'] for tweet in tweets if 'text' in tweet]
lang = [tweet['lang'] for tweet in tweets if 'lang' in tweet]
geo = [tweet['geo'] for tweet in tweets if 'geo' in tweet]
place = [tweet['place'] for tweet in tweets if 'place' in tweet]
# Create a data frame (using pd.Index may be "incorrect", but I am a noob)
df=pd.DataFrame({'Ids':pd.Index(ids),
'Text':pd.Index(text),
'Lang':pd.Index(lang),
'Geo':pd.Index(geo),
'Place':pd.Index(place)})
# Create a data frame satisfying conditions:
df2 = df[(df['Lang']==('en')) & (df['Geo'].dropna())]
So far, everything seems to be working fine.
Now, the extracted values for Geo result in the following example:
df2.loc[1921,'Geo']
{'coordinates': [39.11890951, -84.48903638], 'type': 'Point'}
To get rid of everything except the coordinates inside the squared brackets I tried using:
df2.Geo.str.replace("[({':]", "") ### results in NaN
# and also this:
df2['Geo'] = df2['Geo'].map(lambda x: x.lstrip('{'coordinates': [').rstrip('], 'type': 'Point'')) ### results in syntax error
Please advise on the correct way to obtain coordinates values only.
The following line from your question indicates that this is an issue with understanding the underlying data type of the returned object.
df2.loc[1921,'Geo']
{'coordinates': [39.11890951, -84.48903638], 'type': 'Point'}
You are returning a Python dictionary here -- not a string! If you want to return just the values of the coordinates, you should just use the 'coordinates' key to return those values, e.g.
df2.loc[1921,'Geo']['coordinates']
[39.11890951, -84.48903638]
The returned object in this case will be a Python list object containing the two coordinate values. If you want just one of the values, you can slice the list, e.g.
df2.loc[1921,'Geo']['coordinates'][0]
39.11890951
This workflow is much easier to deal with than casting the dictionary to a string, parsing the string, and recapturing the coordinate values as you are trying to do.
So let's say you want to create a new column called "geo_coord0" which contains all of the coordinates in the first position (as shown above). You could use a something like the following:
df2["geo_coord0"] = [x['coordinates'][0] for x in df2['Geo']]
This uses a Python list comprehension to iterate over all entries in the df2['Geo'] column and for each entry it uses the same syntax we used above to return the first coordinate value. It then assigns these values to a new column in df2.
See the Python documentation on data structures for more details on the data structures discussed above.