Bulk insert rows from an array to an sql server with golang - mysql

I have a list of structs as follows
row = [
{
"name":<name1>,
"age" :<age1>,
"job" :<job1>
},
{
"name":<name1>,
"age" :<age1>,
"job" :<job1>
},
{
"name":<name1>,
"age" :<age1>,
"job" :<job1>
},
etc...
]
I want to insert this into an SQL table. So far I was running a loop through the array and inserting each row one by one. But is there any other way by which I can insert all the rows with just one query? I know bulk insert, but my understanding is that, for bulk insert, I will have to import data from an external file. I don't want to do that. How do I use the data from this array and perform bulk insert?

type Person struct {
Name string
Age int
Job string
}
func InsertPeople(db *sql.DB, personSlice []*Person) error {
var queryString = `INSERT INTO "person_table" (
"name"
, "age"
, "job"
) VALUES `
numOfFields := 3
params := make([]interface{}, len(personSlice)*numOfFields)
for i, p := range personSlice {
pos := i * numOfFields
params[pos+0] = p.Name
params[pos+1] = p.Age
params[pos+2] = p.Job
queryString += `(?, ?, ?),`
}
queryString = queryString[:len(queryString)-1] // drop last comma
_, err := db.Exec(queryString, params...)
return err
}

You aren't going to be able to do any kind of super-optimized bulk insert without placing a file on the server I don't think.
I am not sure if the db library supports it but using the SQLX extension you can build a single insert statement with named bindvars that go against a struct. You can then pass an array of these structs to a method like NamedExec.
Something like this:
users := []User{
{
Name: "alex",
Email: "alex#example.com",
},
{
Name: "muhammed",
Email: "muhammed#example.com",
},
}
db.NamedExec("insert into users (NAME, EMAIL) values (:Name, :Email);", users)

Related

Querying on mysql json array using mysql workbench

Here is my json data:
{
"TransactionId": "1",
"PersonApplicant": [
{
"PersonalId": "1005",
"ApplicantPhone": [
{
"PhoneType": "LANDLINE",
"PhoneNumber": "8085063644",
"IsPrimaryPhone": true
}
]
},
{
"PersonalId": "1006",
"ApplicantPhone": [
{
"PhoneType": "LANDLINE",
"PhoneNumber": "9643645364",
"IsPrimaryPhone": true
},
{
"PhoneType": "HOME",
"PhoneNumber": "987654321",
"IsPrimaryPhone": false
}
]
}
]
}
I want to get phone no of the people who have phonetype as landline.
How to do that?
I tried this approach:
#find phoneNumber when phoneType='LANDLINE'
SELECT
#path_to_name := json_unquote(json_search(applicationData, 'one', 'LANDLINE')) AS path_to_name,
#path_to_parent := trim(TRAILING '.PhoneType' from #path_to_name) AS path_to_parent,
#event_object := json_extract(applicationData, #path_to_parent) as event_object,
json_unquote(json_extract(#event_object, '$.PhoneNumber')) as PhoneNumber
FROM application;
The issue with this is that I am using 'one' so I am able to achieve results but here in my json I have 2 people who have type as landline.
Using json search I am getting array of values and I am not able to decide how to extract these array row values in a manner where I can extract paths.
SELECT
#path_to_name := json_unquote(json_search(applicationData, 'all', 'LANDLINE')) from application;
result:
as you can see at 3rd and 4th row i am getting 2 data as an array.
How do I store this data to get the appropriate result?
I also tried one more query but not able to retrieve results for array of data.
I cannot use stored procedure and I have to use mysql workbench.
Please note that I am fresher so I don't know how I can approach this solution for more complex queries where I may have to retrieve id of a person having type as landline (multiple people in single array).
SELECT test.id, jsontable.*
FROM test
CROSS JOIN JSON_TABLE(test.data,
'$.PersonApplicant[*]'
COLUMNS ( PersonalId INT PATH '$.PersonalId',
PhoneType VARCHAR(255) PATH '$.ApplicantPhone[0].PhoneType',
PhoneNumber VARCHAR(255) PATH '$.ApplicantPhone[0].PhoneNumber')) jsontable
WHERE jsontable.PhoneType = 'LANDLINE';
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=4089207ccfba5068a48e06b52865e759

mySQL/Sequelize - how to query to get data whose field is empty array

My http request will return below data:
It returns below data:
Users.js
{
{
...
friends:[]
},
{
...
friends:[{id:xxx,...},...]
},
{
...
friends:[]
},
}
If I want to use query to get all data whose friends array is [],
should I do below query.
select * from users where (what should I write here)
If friends is a direct column in your database is JSON array. You can use JSON_LENGTH to find out the length of array.
SELECT JSON_LENGTH('[1, 2, {"a": 3}]'); // Output: 3
SELECT JSON_LENGTH('[]'); // Output: 0
You can use same concept to get data from your database.
select *
FROM users
WHERE JSON_LENGTH(friends) = 0;
If you've nested JSON and one of key is friends in that json for given column(data) then your query would be like using JSON_CONTAINS
SELECT *
FROM users
WHERE JSON_CONTAINS(data, JSON_ARRAY(), '$.friends') -- To check do we have `friends` as key in that json
and JSON_LENGTH(data, '$.friends') = 0; -- To check whether it is empty array.
Now you can convert it to sequelize query. One of the way you can use is
Model.findAll({
where: {
[Op.and]: [
Sequelize.literal('RAW SQL STATEMENT WHICH WONT BE ESCAPED!!!')
]
}
})
Make sure to update Model with your user model and query.

U-SQL JsonTuple - How to access specific fields in JSON array

I'm extracting AVRO data which has a JSON field that I need to get values from. The JSON has an array, and I don't know what order the different elements of the array may appear in. How can I target specific node/values?
For example, Filters[0] could be Category one time, but could be AddressType another time.
I'm extracting AVRO data - i.e.
#rs =
EXTRACT date DateTime,
Body byte[]
FROM #input_file
USING new Microsoft.Analytics.Samples.Formats.ApacheAvro.AvroExtractor(#"
...
The Body is JSON that can look like this (but Category is not always Filter[0]. This is a small example; there are 7 different types of "Field"s):
{
""TimeStamp"": ""2019-02-19T15:00:29.1067771-05:00"",
""Filters"": [{
""Operator"": ""eq"",
""Field"": ""Category"",
""Value"": ""Sale""
}, {
""Operator"": ""eq"",
""Field"": ""AddressType"",
""Value"": ""Home""
}
]
}
My U-SQL looks like this, which of course does not always work.
#keyvalues =
SELECT JsonFunctions.JsonTuple(Encoding.UTF8.GetString(Body),
"TimeStamp",
"$.Filters[?(#.Field == 'Category')].Value",
"$.Filters[?(#.Field == 'AddressType')].Value"
) AS message
FROM #rs;
#results =
SELECT
message["TimeStamp"] AS TimeStamp,
message["Filters[0].Value"] AS Category,
message["Filters[1].Value"] AS AddressType
FROM #keyvalues;
Although this does not actually answer my question, as a workaround, I modified the Microsoft 'sample' JsonFunctions.JsonTuple method to be able to specify my own key name for extracted values:
/// Added - Prefix a path expression with a specified key. Use key~$e in the expression.
/// eg:
/// JsonTuple(json, "myId~id", "myName~name") -> field names MAP{ {myId, 1 }, {myName, Ed } }
The modified code:
private static IEnumerable<KeyValuePair<string, T>> ApplyPath<T>(JToken root, string path)
{
var keySeparatorPos = path.IndexOf("~");
string key = null;
var searchPath = path;
if (keySeparatorPos > 0) // =0?if just a leading "=", i.e. no key provided, then don't parse out a key.
{
key = path.Substring(0, keySeparatorPos).Trim();
searchPath = path.Substring(keySeparatorPos + 1);
}
// Children
var children = SelectChildren<T>(root, searchPath);
foreach (var token in children)
{
// Token => T
var value = (T)JsonFunctions.ConvertToken(token, typeof(T));
// Tuple(path, value)
yield return new KeyValuePair<string, T>(key ?? token.Path, value);
}
}
For example, I can access vales and name them
#keyvalues =
SELECT JsonFunctions.JsonTuple(Encoding.UTF8.GetString(Body),
"TimeStamp",
"EventName",
"Plan~ $.UrlParams.plan",
"Category~ $.Filters[?(#.Field == 'Category')].Value",
"AddressType~ $.Filters[?(#.Field == 'AddressType')].Value"
) AS message
FROM #rs;
#results =
SELECT
message["TimeStamp"] AS TimeStamp,
message["EventName"] AS EventName,
message["Plan"] AS Plan,
message["Category"] AS Category,
message["AddressType"] AS AddressType
FROM #keyvalues;
(I've not tested to see what would happen if the same Field appears multiple times in the array; That won't happen in my case)

JSONiq count duplicates in array

I want to count duplicate values with JSONiq. I have following code:
jsoniq version "1.0";
import module namespace fetch = "http://zorba.io/modules/fetch";
let $tweets := parse-json(fetch:content("/tweets.json"))
let $users := parse-json(fetch:content("/users.json"))
return
let $different_languages :=
for $tweet in $tweets[]
return {
"name" : $tweet."metadata"."iso_language_code"
}
return [$different_languages]
This returns all the languages, but it opens a new pair for every language. It looks like this:
[ { "name" : "de" },
{ "name" : "da" },
{ "name" : "da" },
{ "name" : "da" }]
I want to return a JSON object that looks like this:
[ { "count" : 1, "language" : "de" },
{ "count" : 3, "language" : "da" }]
How can i achieve this?
This can be achieved with a group-by clause. This works like a SQL group by, but with an even finer level of control.
In the code below, the four objects in the unboxed $tweets array are grouped according to their language field ($tweet.metadata.iso_language_code). In each evaluation of the return clause, the grouping variable $language will contain the name of the language for the current group, and the non-grouping variable $tweet will contain the sequence of tweets that belong to the group. Calling count() on this sequence will return 3 and 1 respectively.
jsoniq version "1.0";
import module namespace fetch = "http://zorba.io/modules/fetch";
let $tweets := parse-json(fetch:content("/tweets.json"))
let $users := parse-json(fetch:content("/users.json"))
return
for $tweet in $tweets[]
group by $language := $tweet."metadata"."iso_language_code"
return { language: $language, count: count($tweet) }
Also, the quotes on metadata and iso_language_code are not needed. If needed, you can also wrap the result into an array, like so:
jsoniq version "1.0";
import module namespace fetch = "http://zorba.io/modules/fetch";
let $tweets := parse-json(fetch:content("/tweets.json"))
let $users := parse-json(fetch:content("/users.json"))
return [
for $tweet in $tweets[]
group by $language := $tweet.metadata.iso_language_code
return { language: $language, count: count($tweet) }
]
There is no restriction on the expression in the return clause: $language and $tweet are variables like any others and they can be used as inputs to any JSONiq expression. For instance, instead of counting the tweets, one can also nest them in the output, as the data model is arborescent:
return { language: $language, count: [ $tweet ] }

Cassandra query by a field in JSON

I'm using the latest cassandra version and trying to save JSON like below and was successful,
INSERT INTO mytable JSON '{"username": "myname", "country": "mycountry", "userid": "1"}'
Above query saves the record like,
"rows": [
{
"[json]": "{\"userid\": \"1\", \"country\": \"india\", \"username\": \"sai\"}"
}
],
"rowLength": 1,
"columns": [
{
"name": "[json]",
"type": {
"code": 13,
"type": null
}
}
]
Now I would like to retrieve the record based on userid:
SELECT JSON * FROM mytable WHERE userid = fromJson("1") // but this query throws error
All this occurs in a node/express app and I'm using dse-driver as the client driver.
The CQL command worked like below,
SELECT JSON * FROM mytable WHERE userid="1";
However if it has to be executed via the dse-driver then the below snippet worked,
let query = 'SELECT JSON * FROM mytable WHERE userid = ?';
client.execute(query, ["1"], { prepare: true });
where client is,
const dse = require('dse-driver');
const client = new dse.Client({
contactPoints: ['h1', 'h2'],
authProvider: new dse.auth.DsePlainTextAuthProvider('username', 'pass')
});
If your Cassandra version is 2.1x and below, you can use the Python-based approach. Write a python script using Cassandra-Python API
Here you have to get your row first and then use python json's loads method, which will convert your json text column value into JSON object which will be dict in Python. Then you can play around with Python dictionaries and extract your required nested keys. See the below code snippet.
from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider
import json
if __name__ == '__main__':
auth_provider = PlainTextAuthProvider(username='xxxx', password='xxxx')
cluster = Cluster(['0.0.0.0'],
port=9042, auth_provider=auth_provider)
session = cluster.connect("keyspace_name")
print("session created successfully")
rows = session.execute('select * from user limit 10')
for user_row in rows:
#fetchign your json column
column_dict = json.loads(user_row.json_col)
print(column_dict().keys()
Assuming user-id is the partition key, and assuming you want to retrieve a JSON object corresponding to user of id 1, you should try:
SELECT JSON * FROM mytable WHERE userid=1;
If userid is of type text, you will need to add some quotes.