I want to insert records from an API to my database postGIS.
The api geometry respond with a mix of linestring and mulitlinestring with Z value.
How can i mix both types into the same geometry column?
As soon as i hit a record that is a linestring I get an data type error.
If i use geometry type Geometry then I Geometry has Z dimension but column does not.
Here is my table definition. How can I define a table which can mix simple/multilinestring with Z value?
class Item(Base):
__tablename__ = 'test'
id = Column(Integer, primary_key=True)
kategori = Column(String)
geom = Column(Geometry('MultiLineStringZ', dimension=3))
You can force your LineString to be MultiLineString => ST_Multi()
(https://postgis.net/docs/ST_Multi.html)
And, you can also force the geom to be 3D => ST_Force3D()(https://postgis.net/docs/ST_Force_3D.html)
By combining both functions :
ST_Force3D(ST_Multi(my_linestring))
you will achieve what your looking for.
Related
I have a work order in Maximo 7.6.1.1:
The WO has LatitudeY and LongitudeX coordinates in the Service Address tab.
The WO has a custom zone field.
And there is a feature class (polygons) in a separate GIS database.
I want to do spatial query to return an attribute from the polygon record that the WO intersects and use it to populate zone in the WO.
How can I do this?
Related keyword: Maximo Spatial
To do this live in Maximo using an automation script is possible or by writing custom code into Spatial (more challenging). You want to use the /MapServer/identify tool and post the geometry xy, coordinate system, and the layer you want to query. identify window
You will have to format the geometry object correctly and test your post from the window. I usually grab the post from the network section of developer tools once I get it to work and change the output format to json and use it in my code.
You may actually not need to touch your Maximo environment at all. How about just using a trigger on your work orders table ? That trigger can then automatically fill the zone ID from a simple select statement that matches x and y with the zones in the zones table. Here is how that could look like.
This assumes that your work orders are in a table like this:
create table work_orders (
wo_id number primary key,
x number,
y number,
zone_id number
);
and the zones in a table like this
create table zones (
zone_id number primary key,
shape st_geometry
)
Then the trigger would be like this
create or replace trigger work_orders_fill_zone
before insert or update of x,y on work_orders
for each row
begin
select zone_id
into :new.zone_id
from zones
where sde.st_contains (zone_shape, sde.st_point (:new.x, :new.y, 4326) ) = 1;
end;
/
Some assumptions:
The x and y columns contain coordinates in WGS84 longitude/latitude (not in some projection or some other long/lat coordinate system)
Zones don't overlap: a work order point is always therefore in one and only one zone. If not, then the query may return multiple results, which you then need to handle.
Zones fully cover the territory your work orders can take place in. If a work order location can be outside all your zones, then you also need to handle that (the query would return no result).
The x and y columns are always filled. If they are optional, then you also need to handle that case (set zone_id to NULL if either x or y is NULL)
After that, each time a new work order is inserted in the work_orders table, the zone_id column will be automatically updated.
You can initialize zone_id in your existing work orders with a simple update:
update work_orders set x=x, y=y;
This will make the trigger run for each row in the table ... It may take some time to complete if the table is large.
Adapt the code in the Library Scripts section of Maximo 76 Scripting Features (pdf):
#What the script does:
# 1. Takes the X&Y coordinates of a work order in Maximo
# 2. Generates a URL from the coordinates
# 3. Executes the URL via a separate script/library (LIB_HTTPCLIENT)
# 4. Performs a spatial query in an ESRI REST feature service (a separate GIS system)
# 5. Returns JSON text to Maximo with the attributes of the zone that the work
# order intersected
# 6. Parses the zone number from the JSON text
# 7. Inserts the zone number into the work order record
from psdi.mbo import MboConstants
from java.util import HashMap
from com.ibm.json.java import JSONObject
field_to_update = "ZONE"
gis_field_name = "ROADS_ZONE"
def get_coords():
"""
Get the y and x coordinates(UTM projection) from the WOSERVICEADDRESS table
via the SERVICEADDRESS system relationship.
The datatype of the LatitdeY and LongitudeX fields is decimal.
"""
laty = mbo.getDouble("SERVICEADDRESS.LatitudeY")
longx = mbo.getDouble("SERVICEADDRESS.LongitudeX")
#Test values
#laty = 4444444.7001941890
#longx = 666666.0312127020
return laty, longx
def is_latlong_valid(laty, longx):
#Verify if the numbers are legitimate UTM coordinates
return (4000000 <= laty <= 5000000 and
600000 <= longx <= 700000)
def make_url(laty, longx, gis_field_name):
"""
Assembles the URL (including the longx and the laty).
Note: The coordinates are flipped in the url.
"""
url = (
"http://hostname.port"
"/arcgis/rest/services/Example"
"/Zones/MapServer/15/query?"
"geometry={0}%2C{1}&"
"geometryType=esriGeometryPoint&"
"spatialRel=esriSpatialRelIntersects&"
"outFields={2}&"
"returnGeometry=false&"
"f=pjson"
).format(longx, laty, gis_field_name)
return url
def fetch_zone(url):
# Get the JSON text from the feature service (the JSON text contains the zone value).
ctx = HashMap()
ctx.put("url", url)
service.invokeScript("LIBHTTPCLIENT", ctx)
json_text = str(ctx.get("response"))
# Parse the zone value from the JSON text
obj = JSONObject.parse(json_text)
parsed_val = obj.get("features")[0].get("attributes").get(gis_field_name)
return parsed_val
try:
laty, longx = get_coords()
if not is_latlong_valid(laty, longx):
service.log('Invalid coordinates')
else:
url = make_url(laty, longx, gis_field_name)
zone = fetch_zone(url)
#Insert the zone value into the zone field in the work order
mbo.setValue(field_to_update, zone, MboConstants.NOACCESSCHECK)
service.log(zone)
except:
#If the script fails, then set the field value to null.
mbo.setValue(field_to_update, None, MboConstants.NOACCESSCHECK)
service.log("An exception occurred")
LIBHTTPCLIENT: (a reusable Jython library script)
from psdi.iface.router import HTTPHandler
from java.util import HashMap
from java.lang import String
handler = HTTPHandler()
map = HashMap()
map.put("URL", url)
map.put("HTTPMETHOD", "GET")
responseBytes = handler.invoke(map, None)
response = String(responseBytes, "utf-8")
Given there is a dataset of messages, defined by following code:
case class Message(id: Int, value: String)
var messages = Seq(
(0, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}"""),
(1, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}""")
).toDF("id", "value").as[Message]
var schema = new StructType().add("action", StringType).add("timestamp", TimestampType)
var res = messages.select(
from_json(col("value").cast("string"), schema)
)
+------------------------------------+
|jsontostructs(CAST(value AS STRING))|
+------------------------------------+
| [update,2017-10-0...|
| [update,2017-10-0...|
What is the best way to access the schema information in a plain map function. The function itself returns a row which has lost all the Type infos. In order to reach to the values one has to specify the type again e.g
res.head().getStruct(0).getValuesMap[TimestampType](Seq("timestamp"))
=> Map[String,org.apache.spark.sql.types.TimestampType] = Map(timestamp -> 2017-10-06 01:01:19.0)
or
res.head().getStruct(0).getString(0)
=> res20: String = update
Is there some better way to access the raw json data without spark sql aggregation functions?
As a rule of thumb:
To use collection API (map, flatMap, mapPartitions, groupByKey, etc.) use strongly typed API - define record type (case class works the best) which reflects the schema and use Encoders to convert things back and forth:
case class Value(action: String, timestamp: java.sql.Timestamp)
case class ParsedMessage(id: Int, value: Option[Value])
messages.select(
$"id", from_json(col("value").cast("string"), schema).alias("value")
).as[ParsedMessage].map(???)
With Dataset[Row] stay with high level SQL / DataFrame API (select, where, agg, groupBy)
I have a sqlAlchemy model that has one column of type geometry which is defined like this:
point_geom = Column(Geometry('POINT'), index=True)
I'm using geoalchemy2 module:
from geoalchemy2 import Geometry
Then I make my queries using sqlAlchemy ORM, and everything works fine. For example:
data = session.query(myModel).filter_by(...)
My problem is that when I need to get the sql statement of the query object, I use the following code:
sql = data.statement.compile(dialect=postgresql.dialect())
But the column of type geometry is converted to Byte[], so the resulting sql statement is this:
SELECT column_a, column_b, ST_AsBinary(point_geom) AS point_geom
FROM tablename WHERE ...
What should be done to avoid the conversion of the geometry type to byte type?
I had the same problem when was working with Flask-Sqlalchemy and Geoalchemy2 and solved this as follows.
You just need to create a new subclass of GEOMETRY type.
If you look at documentations, the arguments of "GEOMETRY" type are given:
ElementType - which is the type of returned element, by default it's 'WKBElement' (Well-known-binary-element)
as_binary - the function to use, by default it's 'ST_AsEWKB' which in makes a problem on your case
from_text - the geometry constructor used to create, insert and update elements, by default it is 'ST_GeomFromEWKT'
So what I did? I have just created new subclass with required function, element and constructor and used "Geometry" type on my db models as always do.
from geoalchemy2 import Geometry as BaseGeometry
from geoalchemy2.elements import WKTElement
class Geometry(BaseGeometry):
from_text = 'ST_GeomFromText'
as_binary = 'ST_asText'
ElementType = WKTElement
As you can see I have changed only these 3 arguments of a base class.
This will return you a String with required column variables.
It think you can specify that in your query. Something like this:
from geoalchemy2.functions import ST_AsGeoJSON
query = session.query(ST_AsGeoJSON(YourModel.geom_column))
That should change your conversion. There are many conversion functions in the
geoalchemy documentation.
I'm working with a database schema that has a relationship that isn't always true, and I'm not sure how to describe it with sqlalchemy's ORM.
All the primary keys in this database are stored as a blob type, and are 16 byte binary strings.
I have a table called attribute, and this table has a column called data_type. There are a number of built in data_types, that are not defined explicitly in the database. So, maybe a data_type of 00 means it is a string, and 01 means it is a float, etc (those are hex values). The highest value for the built in data types is 12 (18 in decimal).
However, for some rows in attribute, the value of the attribute stored in the row must exist in a pre-defined list of values. In this case, data_type referrs to lookup.lookup_id. The actual data type for the attribute can then be retrieved from lookup.data_type.
I'd like to be able to call just Attribue.data_type and get back 'string' or 'number'. Obviously I'd need to define the {0x00: 'string', 0x01: 'number'} mapping somewhere, but how can I tell sqlalchemy that I want lookup.data_type if the value of attribute.data_type is greater than 18?
There are a couple of ways to do this.
The simplest, by far, is to just put your predefined data types into the table lookup. You say that you "need to define the... mapping somewhere", and a table is as good a place as any.
Assuming that you can't do that, the next simplest thing is to create a python property on class Attribute. The only problem will be that you can't query against it. You'll want to reassign the column data_type so that it maps to _data_type:
data_type_dict = {0x00: 'string',
0x01: 'number,
...}
class Attribute(Base):
__tablename__ = 'attribute'
_data_type = Column('data_type')
...
#property
def data_type(self):
dt = data_type_dict.get(self._data_type, None)
if dt is None:
s = Session.object_session(self)
lookup = s.query(Lookup).filter_by(id=self._data_type).one()
dt = lookup.data_type
return dt
If you want this to be queryable, that is, if you want to be able to do session.query(Attribute).filter_by(data_type='string'), you need to map data_type to something the database can handle, i.e., an SQL statement. You could do this in raw SQL as a CASE expression:
from sqlalchemy.sql.expression import select, case
class Attribute(Base):
...
data_type = column_property(select([attribute, lookup])\
.where(attribute.data_type==lookup.lookup_id)\
.where(case([(attribute.data_type==0x00, 'string'),
(attribute.data_type==0x01, 'number'),
...],
else_=lookup.data_type))
I'm not 100% certain that last part will work; you may need to explicitly join the tables attribute and lookup to specify that it's an outer join, though I think SQLAlchemy does that by default. The downside of this approach is that you are always going to try to join with the table lookup, though to query using SQL, you sort of have to do that.
The final option is to use a polymorphism, and map the two cases (data_type greater/less than 18) to two different subclasses:
class Attribute(Base):
__tablename__ = 'attribute'
_data_type = Column('data_type')
_lookup = column_property(attribute.data_type > 18)
__mapper_args__ = {'polymorphic_on': _lookup}
class FixedAttribute(Attribute):
__mapper_args__ = {'polymorphic_identity': 0}
data_type = column_property(select([attribute.data_type])\
.where(case([(attribute.data_type==0x00, 'string'),
(attribute.data_type==0x01, 'number'),
...])))
class LookupAttribute(Attribute):
__mapper_args__ = {'polymorphic_identity': 1}
data_type = column_property(select([lookup.data_type],
whereclause=attribute.data_type==lookup.lookup_id))
You might have to replace the 'polymorphic_on': _lookup with an explicit attribute.data_type > 18, depending on when that ColumnProperty gets bound.
As you can see, these are all really messy. Do #1 if it's at all possible.
This source details how to use association proxies to create views and objects with values of an ORM object.
However, when I append an value that matches an existing object in the database (and said value is either unique or a primary key), it creates a conflicting object so I cannot commit.
So in my case is this only useful as a view, and I'll need to use ORM queries to retrieve the object to be appended.
Is this my only option or can I use merge (I may only be able to do this if it's a primary key and not a unique constraint), OR set up the constructor such that it will use an existing object in the database if it exists instead of creating a new object?
For example from the docs:
user.keywords.append('cheese inspector')
# Is translated by the association proxy into the operation:
user.kw.append(Keyword('cheese inspector'))
But I'd like to to be translated to something more like: (of course the query could fail).
keyword = session.query(Keyword).filter(Keyword.keyword == 'cheese inspector').one()
user.kw.append(keyword)
OR ideally
user.kw.append(Keyword('cheese inspector'))
session.merge() # retrieves identical object from the database, or keeps new one
session.commit() # success!
I suppose this may not even be a good idea, but it could be in certain use cases :)
The example shown on the documentation page you link to is a composition type of relationship (in OOP terms) and as such represents the owns type of relationship rather then uses in terms of verbs. Therefore each owner would have its own copy of the same (in terms of value) keyword.
In fact, you can use exactly the suggestion from the documentation you link to in your question to create a custom creator method and hack it to reuse existing object for given key instead of just creating a new one. In this case the sample code of the User class and creator function will look like below:
def _keyword_find_or_create(kw):
keyword = Keyword.query.filter_by(keyword=kw).first()
if not(keyword):
keyword = Keyword(keyword=kw)
# if aufoflush=False used in the session, then uncomment below
#session.add(keyword)
#session.flush()
return keyword
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String(64))
kw = relationship("Keyword", secondary=lambda: userkeywords_table)
keywords = association_proxy('kw', 'keyword',
creator=_keyword_find_or_create, # #note: this is the
)
I recently ran into the same problem. Mike Bayer, creator of SQLAlchemy, refered me to the “unique object” recipe but also showed me a variant that uses an event listener. The latter approach modifies the association proxy so that UserKeyword.keyword temporarily points to a plain string and only creates a new Keyword object if the keyword doesn't already exist.
from sqlalchemy import event
# Same User and Keyword classes from documentation
class UserKeyword(Base):
__tablename__ = 'user_keywords'
# Columns
user_id = Column(Integer, ForeignKey(User.id), primary_key=True)
keyword_id = Column(Integer, ForeignKey(Keyword.id), primary_key=True)
special_key = Column(String(50))
# Bidirectional attribute/collection of 'user'/'user_keywords'
user = relationship(
User,
backref=backref(
'user_keywords',
cascade='all, delete-orphan'
)
)
# Reference to the 'Keyword' object
keyword = relationship(Keyword)
def __init__(self, keyword=None, user=None, special_key=None):
self._keyword_keyword = keyword_keyword # temporary, will turn into a
# Keyword when we attach to a
# Session
self.special_key = special_key
#property
def keyword_keyword(self):
if self.keyword is not None:
return self.keyword.keyword
else:
return self._keyword_keyword
#event.listens_for(Session, "after_attach")
def after_attach(session, instance):
# when UserKeyword objects are attached to a Session, figure out what
# Keyword in the database it should point to, or create a new one
if isinstance(instance, UserKeyword):
with session.no_autoflush:
keyword = session.query(Keyword).\
filter_by(keyword=instance._keyword_keyword).\
first()
if keyword is None:
keyword = Keyword(keyword=instance._keyword_keyword)
instance.keyword = keyword