I'm having a use case here which I can't seem to solve. Basically, I need to create a webservice where users may query the couchbase cluster "dynamically". Indeed, i'm storing metadata of different files, and the "creation" of this metadata is up to the user : I don't have specific fields in my Java POJO, i'm inserting a MAP which gets inserted as a nested object in couchbase.
Now the query I need is pretty simple on paper and looks something like this :
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND $1 = $2")
List<FileMetadata> findListMetadata(String pKey, String pValue);
But it doesn't seem to work, $1 doesn't seem to ever get replaced by the pKey variable.
I'm using CouchBase 4.5 with the Spring Data connector.
Any ideas on how to solve that use case ?
You need to dynamically generate the query string, so that pKey is inserted into the query string and pValue is passed as a parameter (as you are doing).
Related
I have a table in AWS Glue, and the crawler has defined one field as array.
The content is in S3 files that have a json format.
The table is TableA, and the field is members.
There are a lot of other fields such as strings, booleans, doubles, and even structs.
I am able to query them all using a simpel query such as:
SELECT
content.my_boolean,
content.my_string,
content.my_struct.value
FROM schema.tableA;
The issue is when I add content.members into the query.
The error I get is: [Amazon](500310) Invalid operation: schema "content" does not exist.
Content exists because i am able to select other fiels from the main key in the json (content).
Probably is something related with how to perform the query agains array field in Spectrum.
Any idea?
You have to rename the table to extract the fields from the external schema:
SELECT
a.content.my_boolean,
a.content.my_string,
a.content.my_struct.value
FROM schema.tableA a;
I had the same issue on my data, I really don't know why it needs this cast but it works. If you need to access elements of an array you have to explod it like:
SELECT member.<your-field>,
FROM schema.tableA a, a.content.members as member;
Reference
You need to create a Glue Classifier.
Select JSON as Classifier type
and for the JSON Path input the following:
$[*]
then run your crawler. It will infer your schema and populate your table with the correct fields instead of just one big array. Not sure if this was what you were looking for but figured I'd drop this here just in case others had the same problem I had.
I am creating a query to re-use periodically to pull data out of a MySQL database (not for use in production code) and want to display values from a JSON object as columns. I installed common_schema and have been using the extract_json function but I can't find the correct xpath to use to get the field I want, I always get null. The query I am using currently is below:
SELECT common_schema.extract_json_value(stores.info,'/Region') as "Sales Region" FROM stores
An example of the JSON object stored in stores.info is below:
{"Town":"HDM","Post Code":"003408","Region":"FGH","OutletCode":"AB43G","CustomerCode":"15134158"}
What xpath do I need to access, for example Region. If the path is correct why is it returning NULL?
Say I have a text field with JSON data like this:
{
"id": {
"name": "value",
"votes": 0
}
}
Is there a way to write a query which would find id and then would increment votes value?
I know i could just retrieve the JSON data update what I need and reinsert updated version, but i wonder is there a way to do this without running two queries?
UPDATE `sometable`
SET `somefield` = JSON_REPLACE(`somefield`, '$.id.votes', JSON_EXTRACT(`somefield` , '$.id.votes')+1)
WHERE ...
Edit
As of MySQL 5.7.8, MySQL supports a native JSON data type that enables efficient access to data in JSON documents.
JSON_EXTRACT will allow you to access a particular JSON element in a JSON field, while JSON_REPLACE will allow you to update it.
To specify the JSON element you wish to access, use a string with the format
'$.[top element].[sub element].[...]'
So in your case, to access id.votes, use the string '$.id.votes'.
The SQL code above demonstrates putting all this together to increment the value of a JSON field by 1.
I think for a task like this you're stuck using a plain old SELECT followed by an UPDATE (after you parse the JSON, increment the value you want, and then serialize the JSON back).
You should wrap these operations in a single transaction, and if you're using InnoDB then you might also consider using SELECT ... FOR UPDATE : http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html
This is sort of a tangent, but I thought I'd also mention that this is the type of operation that a NoSQL database like MongoDB is quite good at.
Situation:
I have user model. attribute "meta_data" in db represents "text" type field.
In model it seriazized by custom class. ( serialize :meta_data, CustomJsonSerializer.new )
It means, when I have an instance of user, I can work with meta_data like with Hash.
User.first.meta_data['username']
Problem:
I need to write a search function, which will search users by given string. I can do it by manual building search query in rails ex. User.where("email LIKE '%#{string}%'")...
But what about meta_data ? Should I search in this field by LIKE statement too? If I will do so, it will decrease relevance of found record.
For example:
I have 2 users. One of them has username "patrick", another one is "sergio"
meta data in db will look like this:
1) {username: patrick}
2) {username: sergio}
I want to find sergio , I enter a search string "ser" => but I have 2 results, instead of one. This meta_data string "{uSERname: Patrick}" also has "ser", so it makes this record irrelevant.
Do you have any idea how to solve it?
That's really the problem with serialized data. In theory, the serialization could be an algorithm that is very unsearchable. It could do a Hoffman encoding, or other compression, and store the serialization in binary. You are relying on the assumption that the serialization uses JSON and your string will still be findable as a sub-string in the serialization.
Then the problem you are having is another issue. Other data in the serialization can mess up your results.
In general, if you serialize data, you are making a choice to not be searchable.
So a solution would be to add an additional field that you populate in a way that you control. Have a values field and store a pipe (|) delimited value that you can search. So if the data is {firstname: "Patrick", lastname: "Stern"}, your meta_values field might be "Patrick|Stern".
Also, don't use the where method with a string with #{} expansion of input values. The makes it vulnerable to SQL attacks. Instead use:
where("meta_values is like :pattern", pattern: "%#{string}%")
I know that may not look very different, but ActiveRecord will go through a sanitizing this way. If someone has a semi-colon in string, then ActiveRecord will escape the semi-colon in the search condition.
I'm transferring a MySQL table to MongoDB. There is a primary key id in my MySQL table and I want this id to be converted to the _id in MongoDB.
I use php's MongoCollection::insert( $mysql_array );
However it doesn't work because if I set $mysql_array['_id'] it is seen by mongo as a String instead of a MongoId. I tried $mysql_array['_id'] = new MongoId( $id ) but it doesn't allow me to override the default _id value. I saw that all my MySQL's integer columns are converted to string by MongoCollection::insert(). If I could set MongoCollection::insert() to correctly transfer an integer it would maybe work.
typecast the _id to a integer value like this...
(int) $mysql_array['_id']
You'll find yourself doing this a lot in mongoDB
The ObjectId is a special type in Mongo, but the _id property doesn't have to be of this type. You can't coerce a string or number into an ObjectId, and you shouldn't.
I assume the problem as you perceive it is that your insert worked, but when you looked at the data in the database the _id property didn't look like _id: ObjectId("1234") (if the original ID was 1234). This is as it should be, and it's perfectly fine.
The idea with ObjectId is that it has a predefined structure that makes it guaranteed (mostly) to be unique across a Mongo cluster, but this also means that it has to have this structure, otherwise it is not an ObjectId.
You also mention that all your integer columns are converted to strings. PHP and PHP libraries, are notoriously sloppy when it comes to types, so make sure that it's not the case that the values are already strings when they come from the MySQL result set. Worst case you have to explicitly cast the values before inserting them into Mongo.
You won't be able to convert an arbitrary String value into an Mongo ObjectId due to its specific characteristics (12 bytes -> 24 chars generated from 4 bytes timestamp, 3 bytes client hostname, 2 bytes PID, 3 bytes inc value).
Either you abandon using the MongoId type in your collection's _id-fields and use your MySQL-ID as a string instead (which is not a problem and makes the most sense) or you let Mongo to generate the documents' _id for you, which is also a suitable solution if you want to be able to use the MongoId functions (assuming you're working with PHP):
The MongoId class
Choosing the second solution you still are able to store your MySQL-IDs in another field of the doc, like id or *mysql_id* to reference them later.
Concerning your question about (int) and (string) values: Are you sure they come as a PHP integer from your MySQL DB? If so, they usually should be stored as integers in Mongo. Check it with a var_dump() and in case of incompatibility cast it with an (int). Maybe it would be helpful if you post your select/insert code...
Use MongoCollection::save() and your array should work.