SQLAlchemy Sessions and other queries - sqlalchemy

Looking a sqlalchemy tutorials, I quite often see querys in the form of
SomeClass.query.filter(...)
And then often with a session object
session.query(SomeClass).count()
What is the deal with the first notation, I thought I would always need a session to retrieve data from the database.


First notation is a shortcut when using Contextual Session API. When using declarative extension it is convenient to specify it on the Base, however one can apply it to any model class without need for declarative.
In order to enable it, one should first set it up using scoped_session.query_property, usually like below:
Session = scoped_session(sessionmaker(bind=engine))
Base.query = Session.query_property()

Related

Bigtable Row Based Access Control Patterns

I would like to implement row based access control for Bigtable using roles so that each row may allow one to many roles. I do not know how I could get this into the key easily.
What patterns are available for implementing something like this?
My current use case is making a prefix range lookup with my other indices as part of the key already.

Unfortunately, currently Bigtable does not support a native row-based ACL feature. It really depends on your use case, but there are several ways to implement this yourself. Following you will find some possible suggestions:
Apache Accumulo is a wrapper around HBase which implements row-based ACLs.
A proxy layer ahead of the table which queries for the "permissions" on the table itself, or a secondary permissions table is viable, however, quite complex to implement yourself. From what I understand, that is the solution you may have gone with, but keep in mind that the caveat is you must implement the check for complex queries as well, which might make it very inefficient (and convoluted to maintain as new features arrive).

roles := []bigtable.Filter{bigtable.ColumnFilter("public"),
bigtable.ColumnFilter("admin")}
acl := bigtable.InterleaveFilters(roles...)
table.ReadRows(context.Background(), rr, func(r bigtable.Row) bool {
// do something
}, bigtable.RowFilter(acl))

What is the difference between SQLAlchemy Core and ORM?

What is the difference of purpose between SQLAlchemy Core and SQLAlchemy ORM?

The ORM is, as the name implies, an object-relational mapper: Its purpose is to represent database relations as Python objects.
The core is a query builder. Its purpose is to provide a programmatic means to generate SQL queries (and DDL) -- but the results of those queries are just tuples (with a little extra magic), not objects of your own ("your" being, in this case, "the application developer's") design.
In general, if you're trying to programmatically build queries (particularly based on information only available at runtime), you should be using the core. If you're trying to build your application MVC-style and want database-backed objects to be the "model", you should be using the ORM.

The SQLAlchemy-ORM is developed on top of SQLAlchemy-Core. As we can see the basic architecture of SQLAlchemy, below.
So, if we want we can use SQLAlchemy-Core to execute the raw SQL queries after creating engine. SQLAlchemy engine object provides a set of methods to perform basic operation like connect(), execute(), etc.
for example
engine = create_engine('mysql://scott:tiger#localhost/test')
connection = engine.connect()
result = connection.execute("select username from users")
for row in result:
print("username:", row['username'])
connection.close()
In case you want the ORM features from SQLAlchemy, you have to use the ORM part. Where you'll be able to create the python Classes which will be treated as table and attributes of the class will be treated as columns. In addition to that, many more methods will be provided by SQLAlchemy to make things easier for Object-relational Mapping.

Unable to use query parameters

I am trying to use the new query parameters to do searches for elements based off child values. Ideally, I want to be able to do something like
https://dinosaur-facts.firebaseio.com/.json?orderBy="hash"&startAt=123&endAt=123
to get a specific element as long as I have given it a unique hash value. In my course to do this, i realized I couldn't do any sorting except for using orderBy="$key". I even went so far as to make a clone of the demo dinosaur-facts data set. I exported the data using the 'export json' button, then imported it into my data set using the 'import json' button, and verified that all the data is the same. I then tried to do the demo queries outlined here, replacing the "dinosaur-facts" with my own domain, and it STILL doesn't work.
When I try
curl https://myapp.firebaseio.com/.json?orderBy="height"
The error I get is:
{"error" : "Index not defined"}
However, if you try
curl https://dinosaur-facts.firebaseio.com/.json?orderBy="height"
You get exactly what you would expect, all the dinosaurs sorted by their height. Is this an issue with my rules? Why can't I do this functionality that is being claimed? Has it not been rolled out to everyone? Do I need to pass my secret token for every one of these? Because when I do that, I get an error saying my auth token could not be parsed. I really have no idea what is happening, I just want to be able to do queries...

To be able to sort on a specific child, there must be an index on that child node. You can add such an index by adding an .indexOn rule to the security/rules in your dashboard, e.g.
".indexOn": ["hash"]
Most client-side APIs that Firebase provides have an implementation that of the ordering/filtering, which allows them to order/filter data even when no index is present. This is handy for development purposes.
But the REST API doesn't have a client-side, so ordering/filtering is only possible after you define the proper index.
See: https://www.firebase.com/docs/security/guide/indexing-data.html:
Indexes are not required for development unless you are using the REST API. The realtime client libraries can execute ad-hoc queries without specifying indexes. Performance will degrade as the data you query grows, so it is important to add indexes before you launch your app if you anticipate querying a large set of data.

Select query to get database objects

I have database which contains huge number of tables, stored procedure. So,
how can i get specific objects like table, stored procedure in a single query for specific database.

SELECT
[schema] = s.name,
[object] = o.name,
o.type_desc
FROM sys.objects AS o
INNER JOIN sys.schemas AS s
ON o.[schema_id] = s.[schema_id]
WHERE o.[type] IN ('P','U');
Some other answers you'll find on this or other sites might suggest some or all of the following:
sysobjects - stay away, this is a backward compatibility view that has been deprecated, and shouldn't be used in any version > SQL Server 2000. See a thorough but not exhaustive replacement map here.
built-in functions like OBJECT_NAME(), SCHEMA_NAME() and OBJECT_SCHEMA_NAME() - I've recommended these myself over the years, until I realized they are blocking functions and don't observe the transaction's isolation semantics. So if you want to grab this information under read uncommitted while there are underlying changes happening, you can't, and you'll have to wait. Which may be what you want to do, but not always.
INFORMATION_SCHEMA - these views are there to satisfy the standards, but aren't complete, are warned to be inaccurate, and aren't updated to reflect new features (I blogged about several specific problems here). So for very basic information (or when you need to write cross-platform metadata code), they may be ok, but in almost all cases I suggest just always using a method you can trust instead of picking and choosing.

Storing Data in MySQL as JSON

I thought this was a n00b thing to do. And, so, I've never done it. Then I saw that FriendFeed did this and actually made their DB scale better and decreased latency. I'm curious if I should do this. And, if so, what's the right way to do it?
Basically, what's a good place to learn how to store everything in MySQL as a CouchDB sort of DB? Storing everything as JSON seems like it'd be easier and quicker (not to build, less latency).
Also, is it easy to edit, delete, etc., things stored as JSON on the DB?

Everybody commenting seems to be coming at this from the wrong angle, it is fine to store JSON code via PHP in a relational DB and it will in fact be faster to load and display complex data like this, however you will have design considerations such as searching, indexing etc.
The best way of doing this is to use hybrid data, for example if you need to search based upon datetime MySQL (performance tuned) is going to be a lot faster than PHP and for something like searching distance of venues MySQL should also be a lot faster (notice searching not accessing). Data you do not need to search on can then be stored in JSON, BLOB or any other format you really deem necessary.
Data you need to access is very easily stored as JSON for example a basic per-case invoice system. They do not benefit very much at all from RDBMS, and could be stored in JSON just by json_encoding($_POST['entires']) if you have the correct HTML form structure.
I am glad you are happy using MongoDB and I hope that it continues to serve you well, but don't think that MySQL is always going to be off your radar, as your app increases in complexity you may well end up needing an RDBMS for some functionality and features (even if it is just for retiring archived data or business reporting)

MySQL 5.7 Now supports a native JSON data type similar to MongoDB and other schemaless document data stores:
JSON support
Beginning with MySQL 5.7.8, MySQL supports a native JSON type. JSON values are not stored as strings, instead using an internal binary format that permits quick read access to document elements. JSON documents stored in JSON columns are automatically validated whenever they are inserted or updated, with an invalid document producing an error. JSON documents are normalized on creation, and can be compared using most comparison operators such as =, <, <=, >, >=, <>, !=, and <=>; for information about supported operators as well as precedence and other rules that MySQL follows when comparing JSON values, see Comparison and Ordering of JSON Values.
MySQL 5.7.8 also introduces a number of functions for working with JSON values. These functions include those listed here:
Functions that create JSON values: JSON_ARRAY(), JSON_MERGE(), and JSON_OBJECT(). See Section 12.16.2, “Functions That Create JSON Values”.
Functions that search JSON values: JSON_CONTAINS(), JSON_CONTAINS_PATH(), JSON_EXTRACT(), JSON_KEYS(), and JSON_SEARCH(). See Section 12.16.3, “Functions That Search JSON Values”.
Functions that modify JSON values: JSON_APPEND(), JSON_ARRAY_APPEND(), JSON_ARRAY_INSERT(), JSON_INSERT(), JSON_QUOTE(), JSON_REMOVE(), JSON_REPLACE(), JSON_SET(), and JSON_UNQUOTE(). See Section 12.16.4, “Functions That Modify JSON Values”.
Functions that provide information about JSON values: JSON_DEPTH(), JSON_LENGTH(), JSON_TYPE(), and JSON_VALID(). See Section 12.16.5, “Functions That Return JSON Value Attributes”.
In MySQL 5.7.9 and later, you can use column->path as shorthand for JSON_EXTRACT(column, path). This works as an alias for a column wherever a column identifier can occur in an SQL statement, including WHERE, ORDER BY, and GROUP BY clauses. This includes SELECT, UPDATE, DELETE, CREATE TABLE, and other SQL statements. The left hand side must be a JSON column identifier (and not an alias). The right hand side is a quoted JSON path expression which is evaluated against the JSON document returned as the column value.
See Section 12.16.3, “Functions That Search JSON Values”, for more information about -> and JSON_EXTRACT(). For information about JSON path support in MySQL 5.7, see Searching and Modifying JSON Values. See also Secondary Indexes and Virtual Generated Columns.
More info:
https://dev.mysql.com/doc/refman/5.7/en/json.html

CouchDB and MySQL are two very different beasts. JSON is the native way to store stuff in CouchDB. In MySQL, the best you could do is store JSON data as text in a single field. This would entirely defeat the purpose of storing it in an RDBMS and would greatly complicate every database transaction.
Don't.
Having said that, FriendFeed seemed to use an extremely custom schema on top of MySQL. It really depends on what exactly you want to store, there's hardly one definite answer on how to abuse a database system so it makes sense for you. Given that the article is very old and their main reason against Mongo and Couch was immaturity, I'd re-evaluate these two if MySQL doesn't cut it for you. They should have grown a lot by now.

json characters are nothing special when it comes down to storage, chars such as
{,},[,],',a-z,0-9.... are really nothing special and can be stored as text.
the first problem your going to have is this
{
profile_id: 22,
username: 'Robert',
password: 'skhgeeht893htgn34ythg9er'
}
that stored in a database is not that simple to update unless you had your own proceedure and developed a jsondecode for mysql
UPDATE users SET JSON(user_data,'username') = 'New User';
So as you cant do that you would Have to first SELECT the json, Decode it, change it, update it, so in theory you might as well spend more time constructing a suitable database structure!
I do use json to store data but only Meta Data, data that dont get updated often, not related to the user specific.. example if a user adds a post, and in that post he adds images ill parse the images and create thumbs and then use the thumb urls in a json format.

To illustrate how difficult it is to get JSON data using a query, I will share the query I made to handle this.
It doesn't take into account arrays or other objects, just basic datatypes. You should change the 4 instances of column to the column name storing the JSON, and change the 4 instances of myfield to the JSON field you want to access.
SELECT
SUBSTRING(
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', ''),
LOCATE(
CONCAT('myfield', ':'),
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', '')
) + CHAR_LENGTH(CONCAT('myfield', ':')),
LOCATE(
',',
SUBSTRING(
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', ''),
LOCATE(
CONCAT('myfield', ':'),
REPLACE(REPLACE(REPLACE(column, '{', ''), '}', ','), '"', '')
) + CHAR_LENGTH(CONCAT('myfield', ':'))
)
) - 1
)
AS myfield
FROM mytable WHERE id = '3435'

This is an old question, but I am still able to see this at the top of the search result of Google, so I guess it would be meaningful to add a new answer 4 years after the question is asked.
First of all, there is better support in storing JSON in RDBMS. You may consider switching to PostgreSQL (although MySQL has supported JSON since v5.7.7). PostgreSQL uses very similar SQL commands as MySQL except they support more functions. One of the functions they added is that they provide JSON data type and you are now able to query the JSON stored. (Some reference on this) If you are not making up the query directly in your program, for example, using PDO in php or eloquent in Laravel, all you need to do is just to install PostgreSQL on your server and change database connection settings. You don't even need to change your code.
Most of the time, as the other answers suggested, storing data as JSON directly in RDBMS is not a good idea. There are some exception though. One situation I can think of is a field with variable number of linked entry.
For example, for storing tag of a blog post, normally you will need to have a table for blog post, a table of tag and a matching table. So, when the user wants to edit a post and you need to display which tag is related to that post, you will need to query 3 tables. This will damage the performance a lot if your matching table / tag table is long.
By storing the tags as JSON in the blog post table, the same action only requires a single table search. The user will then be able to see the blog post to be edit quicker, but this will damage the performance if you want to make a report on what post is linked to a tag, or maybe search by tag.
You may also try to de-normalize the database. By duplicating the data and storing the data in both ways, you can receive benefit of both method. You will just need a little bit more time to store your data and more storage space (which is cheap comparing to the cost of more computing power)

It really depends on your use case. If you are storing information that has absolutely no value in reporting, and won't be queried via JOINs with other tables, it may make sense for you to store your data in a single text field, encoded as JSON.
This could greatly simplify your data model. However, as mentioned by RobertPitt, don't expect to be able to combine this data with other data that has been normalized.

I would say the only two reasons to consider this are:
performance just isn't good enough with a normalised approach
you cannot readily model your particularly fluid/flexible/changing data
I wrote a bit about my own approach here:
What scalability problems have you encountered using a NoSQL data store?
(see the top answer)
Even JSON wasn't quite fast enough so we used a custom-text-format approach. Worked / continues to work well for us.
Is there a reason you're not using something like MongoDB? (could be MySQL is "required"; just curious)

Here is a function that would save/update keys of a JSON array in a column and another function that retrieves JSON values. This functions are created assuming that the column name of storing the JSON array is json. It is using PDO.
Save/Update Function
function save($uid, $key, $val){
global $dbh; // The PDO object
$sql = $dbh->prepare("SELECT `json` FROM users WHERE `id`=?");
$sql->execute(array($uid));
$data = $sql->fetch();
$arr = json_decode($data['json'],true);
$arr[$key] = $val; // Update the value
$sql=$dbh->prepare("UPDATE `users` SET `json`=? WHERE `id`=?");
$sql->execute(array(
json_encode($arr),
$uid
));
}
where $uid is the user's id, $key - the JSON key to update and it's value is mentioned as $val.
Get Value Function
function get($uid, $key){
global $dbh;
$sql = $dbh->prepare("SELECT `json` FROM `users` WHERE `id`=?");
$sql->execute(array($uid));
$data = $sql->fetch();
$arr = json_decode($data['json'], true);
return $arr[$key];
}
where $key is a key of JSON array from which we need the value.

It seems to me that everyone answering this question is kind-of missing the one critical issue, except #deceze -- use the right tool for the job. You can force a relational database to store almost any type of data and you can force Mongo to handle relational data, but at what cost? You end up introducing complexity at all levels of development and maintenance, from schema design to application code; not to mention the performance hit.
In 2014 we have access to many database servers that handle specific types of data exceptionally well.
Mongo (document storage)
Redis (key-value data storage)
MySQL/Maria/PostgreSQL/Oracle/etc (relational data)
CouchDB (JSON)
I'm sure I missed some others, like RabbirMQ and Cassandra. My point is, use the right tool for the data you need to store.
If your application requires storage and retrieval of a variety of data really, really fast, (and who doesn't) don't shy away from using multiple data sources for an application. Most popular web frameworks provide support for multiple data sources (Rails, Django, Grails, Cake, Zend, etc). This strategy limits the complexity to one specific area of the application, the ORM or the application's data source interface.

Early support for storing JSON in MySQL has been added to the MySQL 5.7.7 JSON labs release (linux binaries, source)! The release seems to have grown from a series of JSON-related user-defined functions made public back in 2013.
This nascent native JSON support seems to be heading in a very positive direction, including JSON validation on INSERT, an optimized binary storage format including a lookup table in the preamble that allows the JSN_EXTRACT function to perform binary lookups rather than parsing on every access. There is also a whole raft of new functions for handling and querying specific JSON datatypes:
CREATE TABLE users (id INT, preferences JSON);
INSERT INTO users VALUES (1, JSN_OBJECT('showSideBar', true, 'fontSize', 12));
SELECT JSN_EXTRACT(preferences, '$.showSideBar') from users;
+--------------------------------------------------+
| id | JSN_EXTRACT(preferences, '$.showSideBar') |
+--------------------------------------------------+
| 1 | true |
+--------------------------------------------------+
IMHO, the above is a great use case for this new functionality; many SQL databases already have a user table and, rather than making endless schema changes to accommodate an evolving set of user preferences, having a single JSON column a single JOIN away is perfect. Especially as it's unlikely that it would ever need to be queried for individual items.
While it's still early days, the MySQL server team are doing a great job of communicating the changes on the blog.

JSON is a valid datatype in PostgreSQL database as well. However, MySQL database has not officially supported JSON yet. But it's baking: http://mysqlserverteam.com/json-labs-release-native-json-data-type-and-binary-format/
I also agree that there are many valid cases that some data is better be serialized to a string in a database. The primary reason might be when it's not regularly queried, and when it's own schema might change - you don't want to change the database schema corresponding to that. The second reason is when the serialized string is directly from external sources, you may not want to parse all of them and feed in the database at any cost until you use any. So I'll be waiting for the new MySQL release to support JSON since it'll be easier for switching between different database then.

I know this is really late but I did have a similar situation where I used a hybrid approach of maintaining RDBMS standards of normalizing tables upto a point and then storing data in JSON as text value beyond that point. So for example I store data in 4 tables following RDBMS rules of normalization. However in the 4th table to accomodate dynamic schema I store data in JSON format. Every time I want to retrieve data I retrieve the JSON data, parse it and display it in Java. This has worked for me so far and to ensure that I am still able to index the fields I transform to json data in the table to a normalized manner using an ETL. This ensures that while the user is working on the application he faces minimal lag and the fields are transformed to a RDBMS friendly format for data analysis etc. I see this approach working well and believe that given MYSQL (5.7+) also allows parsing of JSON this approach gives you the benefits of both RDBMS and NOSQL databases.

I use json to record anything for a project, I use three tables in fact ! one for the data in json, one for the index of each metadata of the json structure (each meta is encoded by an unique id), and one for the session user, that's all.
The benchmark cannot be quantified at this early state of code, but for exemple I was user views (inner join with index) to get a category (or anything, as user, ...), and it was very slow (very very slow, used view in mysql is not the good way).
The search module, in this structure, can do anything I want, but, I think mongodb will be more efficient in this concept of full json data record.
For my exemple, I user views to create tree of category, and breadcrumb, my god ! so many query to do ! apache itself gone ! and, in fact, for this little website, I use know a php who generate tree and breadcrumb, the extraction of the datas is done by the search module (who use only index), the data table is used only for update.
If I want, I can destroy the all indexes, and regenerate it with each data, and do the reverse work to, like, destroy all the data (json) and regenerate it only with the index table.
My project is young, running under php and mysql, but, sometime I thing using node js and mongodb will be more efficient for this project.
Use json if you think you can do, just for do it, because you can ! and, forget it if it was a mistake; try by make good or bad choice, but try !
Low
a french user

I believe that storing JSON in a mysql database does in fact defeat the purpose of using RDBMS as it is intended to be used. I would not use it in any data that would be manipulated at some point or reported on, since it not only adds complexity but also could easily impact performance depending on how it is used.
However, I was curious if anyone else thought of a possible reason to actually do this. I was thinking to make an exception for logging purposes. In my case, I want to log requests that have a variable amount of parameters and errors. In this situation, I want to use tables for the type of requests, and the requests themselves with a JSON string of different values that were obtained.
In the above situation, the requests are logged and never manipulated or indexed within the JSON string field. HOWEVER, in a more complex environment, I would probably try to use something that has more of an intention for this type of data and store it with that system. As others have said, it really depends on what you are trying to accomplish, but following standards always helps longevity and reliability!

You can use this gist: https://gist.github.com/AminaG/33d90cb99c26298c48f670b8ffac39c3
After installing it to the server (just need root privilege not super), you can do something like this:
select extract_json_value('{"a":["a","2"]}','(/a)')
It will return
a 2
.You can return anything inside JSON by using this
The good part is that it is support MySQL 5.1,5.2,5.6. And you do not need to install any binary on the server.
Based on old project common-schema, but it is still working today
https://code.google.com/archive/p/common-schema/