How do I print it into an organised json file? - json

I need help with the output of my json file. I'm trying to print out the keys in a list called keypairs. I have to generate 60 keys, which i have included in the (count<60) part of my code (which isnt here). Im just showing the exporting part of my codes which i have a problem with. Here are my codes:
with open("/home/pi/Desktop/database.json", 'w+'):
db = TinyDB('/home/pi/Desktop/database.json')
table = db.table('Books')
db.insert({'Book ID' : keypair[bookid], 'Serial No.' : keypair[bookserial] })
However, the problem I have right now is that it prints out all the keys in the lists - bookid and bookserial instead of printing a pair of keys in one { }.
Here's an example output where count<2:
Pillar": {}, "_default":
{"1":
{"Bookid": ["b'\\XXXXXX bookid 1 XXXXXXX'", "b'\\AAAAAA bookid 2 AAAAAAA'"],
"Serial No.": ["b'\\YYYYYserialno 1YYYYY'", "b'BBBBserial no2BBBB'"]
}
}
This is the output that i intend to get:
in a format like this where a pair of keys are printed everytime it runs again:
{ Books:
{
Book ID: XXXX bookid 1 XXXX
Serial No.: XXX serialno 1 XXXX
}
{
Book ID: XXX bookid 2 XXXX
Serial No.: XXX serial no. 2 XXX
}
}
As you can see, the json file is messy, i have no idea how to make it automatically neat and also, you can see that the keys print out together into one instead of separately. Imagine having to print 60 pairs of these and they are all under one { }.
Help!

I see multiple syntax problems in your code when you are trying to create the json. You could try to do this.
{ Books:
{
{
Book ID: XXXXXXXX
Serial No.: XXXXXXX
},
{
Book ID: XXXXXXX
Serial No.: XXXXXX
}
}
}
As you can see i wrapped the book items in brackets as well and the items are comma separated now.
This could help you with your problem.

Related

Reshape JSON with jq to expand each object into multiple rows

I have a database of resumes in json format that I want to reshape so that each row corresponds to a person's employment history at a given company:
personid, company_name, start_date, end_date
However, running the following jq command
{personid:.personid, company_name: .experience[].company.name, sdate: .experience[].start_date, edate: .experience[].end_date}
produces the cartesian product of all the fields (3 jobs x 3 fields). For example, a person who has held 3 jobs at 3 different companies in the past looks like this after running the jq command above:
{"id":"abc123","companyname":"companyA","sdate":"2020-06","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2020-06","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2020-06","edate":"2017-07"}
{"id":"abc123","companyname":"companyA","sdate":"2016-10","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2016-10","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2016-10","edate":"2017-07"}
{"id":"abc123","companyname":"companyA","sdate":"2017-05","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2017-05","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2017-05","edate":"2017-07"}
There are 9 entries for CompanyB and CompanyC each but I truncated the output above for brevity.
I think I need to use the group_by() command, but I've been unsuccessful.
Thanks in advance.
Without seeing the original data, my guess is that you get the cartesian product because you are iterating three times (.experience[]) within the object construction. You might want to pull out the iteration, maybe save it in a variable, and reference that instead:
.experience[] as $experience | {
personid: .personid,
company_name: $experience.company.name,
sdate: $experience.start_date,
edate: $experience.end_date
}
Depending on the outer structure of your construction, also the other way around may be appropriate, ie. storing the .person field instead in a variable:
.personid as $id | .experience[] | {
personid: $id,
company_name: .company.name,
sdate: .start_date,
edate: .end_date
}

Node JS How to change key names in an object using another object for conversion?

I have these tables on one DB (MySQL) which I need to sync with corresponding tables on another DB (MSSQL), but the field names are different. I was wondering what efficient way there is to convert the names of the fields after fetching the rows so that I could insert them into the other tables.
I was thinking of doing something like this. Make objects where the key is the original table column's names and the value is the destination table column's names:
{
name : UNAME
id : CID
location : LOC
}
And the rows that I fetched and need to insert would look something like this:
{
name: Ethan
id: 1234
location: somewhere1
},
{
name: Jhon
id: 5678
location: somewhere2
}
and then run on these objects and change their key names according to the conversion table, so that I can insert them to the destination table properly.
I can't just insert without field names, as the fields are not in the same order.
How can I do what I've described efficiently? Do you have ideas for better strategies to accomplish this?
thanks a lot!
Sounds about right, how about this:
const converter = {
name : UNAME
id : CID
location : LOC
}
let newData = []
dbResults.forEach(row => {
newData.push({
name: `${row[converter['name']]}`
id: `${row[converter['id']]}`
location: `${row[converter['location']]}`
})
})
EDIT:
Actually looking at the above code there is no need for the converter object:
let newData = []
dbResults.forEach(row => {
newData.push({
name: `${row['UNAME']}`
id: `${row['CID']}`
location: `${row['LOC]}`
})
})

Is there a way to enrich JSON field in MySQL?

Let's take a simple schema with two tables, one that describes an simple entity item (id, name)
id | name
------------
1 | foo
2 | bar
and another, lets call it collection, that references to an item, but inside a JSON Object in something like
{
items: [
{
id: 1,
quantity: 2
}
]
}
I'm looking for a way to eventually enrich this field (kind of like populate in Mongo) in the collection with the item element referenced, to retrieve something like
{
...
items: [
{
item: {
id: 1,
name: foo
},
quantity: 2
}
]
...
}
If you have a solution with PostgreSQL, I take it as well.
If I understood correctly, your requirement is to convert an Input JSON data into MySQL table so that you can work with JSON but leverage the power of SQL.
Mysql8 recently released JSONTABLE function. By using this function, you can store your JSON in the table directly and then query it like any other SQL query.
It should serve your immediate case, but this means that your table schema will have a JSON column instead of traditional MySQL columns. You will need to check if it serves your purpose.
This is a good tutorial for the same.

Json path extraction with filters

I need to get the path of the element - list all the libraryphone numbers
I have the following json structure:
{
library:[{
libraryphone: 9898989898,
location:{
address:{
street:123 garden ave,
city: Dublin
}
},
{
libraryphone: 9090909090,
location:{...}
}, {
libraryphone: 9797979797,
location: {...}
}
}]
}
In the json above, I have multiple library phones and addresses. So I need to collect all the phone numbers on the condition that the city exists.
I have tried:
$.library[?(#.location.address.city)].libraryphone
I understand to get the libraryphones, all I need to do is
$.library[*].libraryphone
But, I need to have the condition of checking if the city exists or not.

JSON path parent object, or equivalent MongoDB query

I am selecting nodes in a JSON input but can't find a way to include parent object detail for each array entry that I am querying. I am using pentaho data integration to query the data using JSON input form a mongodb input.
I have also tried to create a mongodb query to achieve the same but cannot seem to do this either.
Here are the two fields/paths that display the data:
$.size_break_costs[*].size
$.size_break_costs[*].quantity
Here is the json source format:
{
"_id" : ObjectId("4f1f74ecde074f383a00000f"),
"colour" : "RAVEN-SMOKE",
"name" : "Authority",
"size_break_costs" : [
{
"quantity" : NumberLong("80"),
"_id" : ObjectId("518ffc0697eee36ff3000002"),
"size" : "S"
},
{
"quantity" : NumberLong("14"),
"_id" : ObjectId("518ffc0697eee36ff3000003"),
"size" : "M"
},
{
"quantity" : NumberLong("55"),
"_id" : ObjectId("518ffc0697eee36ff3000004"),
"size" : "L"
}
],
"sku" : "SK3579"
}
I currently get the following results:
S,80
M,14
L,55
I would like to get the SKU and Name as well as my source will have multiple products (SKU/Description):
SK3579,Authority,S,80
SK3579,Authority,M,14
SK3579,Authority,L,55
When I try To include using $.sku, I the process errors.
The end result i'm after is a report of all products and the available quantities of their various sizes. Possibly there's an alternative mongodb query that provides this.
EDIT:
It seems the issue may be due to the fact that not all lines have the same structure. For example the above contains 3 sizes - S,M,L. Some products come in one size - PACK. Other come in multiple sizes - 28,30,32,33,34,36,38 etc.
The error produced is:
*The data structure is not the same inside the resource! We found 1 values for json path [$.sku], which is different that the number retourned for path [$.size_break_costs[].quantity] (7 values). We MUST have the same number of values for all paths.
I have tried the following mongodb query separately which gives the correct results, but the corresponding export of this doesn't work. No values are returned for the Size and Quantity.
Query:
db.product_details.find( {}, {sku: true, "size_break_costs.size": true, "size_break_costs.quantity": true}).pretty();
Export:
mongoexport --db brandscope_production --collection product_details --csv --out Test01.csv --fields sku,"size_break_costs.size","size_break_costs.quantity" --query '{}';
Shortly after I added my own bounty, I figured out the solution. My problem has the same basic structure, which is a parent identifier, and some number N child key/value pairs for ratings (quality, value, etc...).
First, you'll need a JSON Input step that gets the SKU, Name, and size_break_costs array, all as Strings. The important part is that size_break_costs is a String, and is basically just a stringified JSON array. Make sure that under the Content tab of the JSON Input, that "Ignore missing path" is checked, in case you get one with an empty array or the field is missing for some reason.
For your fields, use:
Name | Path | Type
ProductSKU | $.sku | String
ProductName | $.name | String
SizeBreakCosts | $.size_break_costs | String
I added a "Filter rows" block after this step, with the condition "SizeBreakCosts IS NOT NULL", which is then passed to a second JSON Input block. This second JSON block, you'll need to check "Source is defined in a field?", and set the value of "Get source from field" to "SizeBreakCosts", or whatever you named it in the first JSON Input block.
Again, make sure "Ignore missing path" is checked, as well as "Ignore empty file". From this block, we'll want to get two fields. We'll already have ProductSKU and ProductName with each row that's passed in, and this second JSON Input step will further split it into however many rows are in the SizeBreakCosts input JSON. For fields, use:
Name | Path | Type
Quantity | $.[*].quantity | Integer
Size | $.[*].size | String
As you can see, these paths use "$.[*].FieldName", because the JSON string we passed in has an array as the root item, so we're getting every item in that array, and parsing out its quantity and size.
Now every row should have the SKU and name from the parent object, and the quantity and size from each child object. Dumping this example to a text file, I got:
ProductSKU;ProductName;Size;Quantity
SK3579;Authority;S; 80
SK3579;Authority;M; 14
SK3579;Authority;L; 55