Concatenate multiple PostreSQL JSON columns in a single JSON root object - json

I have a table (preferences), with several JSON columns (design_prefs, country_prefs, view_prefs, payment_prefs, etc.)
design_prefs --> {"color":"red", "height":15}
country_prefs --> {"language":"en", "currency":"pound"}
view_prefs --> {"layout":"list", "sortby":"category1", "view_brand":1, "view_price":1}
etc.
I retrieve a single row, but only with the columns required by the web context. Maybe just design_prefs, or design_prefs AND country_prefs, or design_prefs AND payment_prefs or whatever combination.
In the application (using Perl, Mojolicious and Mojo::Pg) I convert each JSON column in a Hash (key/value dictionary object used in Perl). And then I concatenate the multiple resulting hashes in a big "preferences" hash object.
It is working, but I would like to do the concatenation in Postgres, so in Mojolicious I only need to do a "JSON to hash" process.
Looking for aggregation functions in Postgresql I have found ways to do something like this:
{"color":"red", "height":15 }, {"language":"en", "currency":"pound"}
But I need this:
{"color":"red", "height":15, "language":"en", "currency":"pound"}
JSON type is stored as text in Pg 9.3, including the initial and ending bracket characters, so direct text concatenation is not a way.
Any guidance is appreciated, even recommendation to forget the idea and keep using the application instead the database.

Related

Splunk: How to extract fields directly in search bar without having to use regular expressions?

I find it very hard to use regular expressions directly in the search bar to extract fields. Another problem is that I do not have the permission to share my extracted fields (extracted by the field extractor and stated in field extractions) with other people. I am now looking for another way to extract fields directly in the search bar. Is there something like this possible in Splunk?
Thanks!
Regular expressions aren't so bad once you've had some practice. Think of them as another programming language to know.
There are other ways to extract fields, but most are less efficient and all are less flexible.
The spath and xpath commands will extract fields from JSON and XML, respectively.
multikv extracts fields from table-formatted data (like from top).
The extract command can be used to parse key/value pairs into fields.
The eval command can be used in combination with various functions to parse events into fields.

Remove JSON keys with wildcards from a MySQL field

I have a MySQL 8.0.20 database with a table that describes metadata about uploaded image files. One column contains a JSON object with a whole bunch of auto-generated data that I'm trying to clean up.
This JSON object sometimes contains one or more variable key names that match a specific pattern. Something like
{
"image_name": "P10043983",
"image_size": "60138",
"image_original_exifdata": "{
'FileName':'P10043983.jpg',
'MimeType':'image/jpeg',
'UndefinedTag:0xA435':'\u0000\u0000\u0000\u0000\u0000\u0000'
}"
}
That UndefinedTag:0xA435 (with many permutations) is the problem. It's referring to various image Exif details like lens type, GPS data, etc. It's stuff that I'm not interested in and that these cameras mostly don't provide, so I've ended up with a table full of long strings of useless characters just taking up space. I want those JSON fields gone for performance and cleanliness.
Is there a way to run a SQL query that would use wildcards or regular expressions to find (and, ideally, remove) all of these pesky variable keys? I'd like to avoid manually making a list of all of the possible "UndefinedTag" keys to search against, and I also didn't like the results when I just treated the whole thing as a string and did REGEXP_REPLACE calls (it sometimes left trailing commas that broke my JSON and were difficult for me to avoid/resolve).
I know some of the JSON functions like JSON_SEARCH() accept wildcards, but it explicitly says the search path can't end in a wildcard (so no UndefinedTag:0x** allowed). Many of the functions I'm after (e.g., JSON_REMOVE()) don't accept wildcards at all. Hell, I've even had trouble finding known keys, and I suspect that silly colon in the key name might have something to do with it.
So, how can I clean up my table and remove the many forms of this UndefinedTag problem? Maybe it's easier to just go back to the regex_replace plan and deal instead with the trailing commas?

What's the difference between json data being encoded or not

What's the purpose (not what it becomes) of doing json_encode on this before I am putting into the database
rating: {cleanliness: 3, publicFacility: 1, roomFacility: 2, security: 2}
to become this
rating: "{"cleanliness":3,"publicFacility":1,"roomFacility":2,"security":2}"
I see no point of doing this cause I need to json_decode it again before serving it back... can anybody clear me out?
Do not store json encoded data in the database. You mitigate the whole point of a relational database this way and make searching for values an expensive task. I see in your sample the attributes cleanliness, publicFacility, roomFacility and security. Those should be columns in your database so you can search for something like "all entries with a cleanliness higher than 3".
It works with the JSON column type but it is more expensive than using normal columns.
Edit: Check the use-case for your database entry. If you are sure you never need to search in or order by the encoded attributes you can store data encoded as json string. However, if your database supports the JSON column type, you should use that one because it allows searching in the stored JSON (but is more expensive than searching in normal columns). </Edit>
Second point: The second code snipped (with the quotation marks) looks like invalid syntax for json.

How do I identify this JSON-like data structure?

I just came across a JSON wannabe that decides to "improve" it by adding datatypes... of course, the syntax makes it nearly impossible to google.
a:4:{
s:3:"cmd";
s:4:"save";
s:5:"token";
s:22:"5a7be6ad267d1599347886";
}
Full data is... much larger...
The first letter seems to be a for array, s for string, then the quantity of data (# of array items or length of string), then the actual piece of data.
With this type of syntax, I currently can't Google meaningful results. Does anyone recognize what god-forsaken language or framework this is from?
Note: some genius decided to stuff this data as a single field inside a database, and it included critical fields that I need to perform aggregate functions on. The rest I can handle if I can get a way to parse this data without resorting to ugly serial processing.
If this can be parsed using MSSQL 2008 that results in a view, I'll throw in a bounty...
I would parse it with a UDF written in .NET - https://learn.microsoft.com/en-us/sql/relational-databases/clr-integration-database-objects-user-defined-functions/clr-user-defined-functions
You can either write a custom aggregate function to parse and calculate these nutty fields, or a scalar value function that returns the field as JSON.
I'd probably opt for the latter in the name of separation of concerns.

MySQL UDF for working with json?

Are there any good UDFs in MySQL to deal with json data, that supports the ability to retrieve a particular value in json (by dot notation key - EG: json_get('foo.bar.baz')) as well as the ability to set the value of a particular key - EG: json_set('foo.bar.baz', 'value')?
I found http://www.mysqludf.org/lib_mysqludf_json/ - but it seems to only provide the ability to create json data structures from non-json column values, as opposed to interacting with json column values.
This UDF is able to parse JSON and return the value of an attribute:
https://github.com/kazuho/mysql_json
This other one too: https://github.com/webaroo/mysql-json-udf