Bulk importing JSON into SQL Server - json

I need to bulk import data into a SQL Server database.
My JSON looks like this:
$projectsJSON = #"
[
{
"id": 35,
"created_at": "2016-01-12T11:40:36+01:00",
"customer_id": 34,
"name": ".com",
"note": "COMXXXX-",
"updated_at": "2016-07-15T12:13:54+02:00",
"archived": false,
"customer_name": "PMName"
},
{
"id": 23,
"created_at": "2010-01-11T12:58:50+01:00",
"customer_id": 43,
"name": "PN",
"note": "{\r\n \"Billable\": 1\r\n}\r\n",
"updated_at": "2017-11-24T15:49:31+01:00",
"archived": false,
"customer_name": "MSM"
}
]
"#
$projects = $projectsJSON |ConvertFrom-Json
$dt2 = New-Object system.Data.DataTable
$dt2 = $projects|select-object id, created_at, customer_id, name, note, updated_at, archived, customer_name
$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity)
$bulkCopy.BulkCopyTimeout = 600
$bulkCopy.DestinationTableName = "project"
$bulkCopy.WriteToServer($dt2)
Unfortunatelly, I keep getting this error:
Cannot convert argument "rows", with value: "System.Object[]", for
"WriteToServer" to type "System.Data.DataRow[]": "Cannot convert the
"#{id=35; created_at=2016-01-12T11:40:36+01:00; customer_id=34;
name=.com; note=COMXXXX-; updated_at=2016-07-15T12:13:54+02:00;
archived=False; customer_name=PMName}" value of type
"Selected.System.Management.Automation.PSCustomObject" to type
"System.Data.DataRow"." At P:\PsideSync\sync.ps1:261 char:3
+ $bulkCopy.WriteToServer($dt2)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodException
+ FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument
What would be the right way to import lots of JSON Data into SQL Server?
TIA

As #IRon notes you are filling $dt2 twice. I assume you want to go with the second initialization.
You could get Out-DataTable from here: https://gallery.technet.microsoft.com/scriptcenter/4208a159-a52e-4b99-83d4-8048468d29dd
Then it's just a matter of calling
$projects = $projectsJSON |ConvertFrom-Json
$dt2 = $projects|select-object id, created_at, customer_id, name, note, updated_at, archived, customer_name
$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity)
$bulkCopy.BulkCopyTimeout = 600
$bulkCopy.DestinationTableName = "project"
$bulkCopy.WriteToServer($dt2 | Out-DataTable)
Disclaimer: I haven't tried this myself.

Related

Exploding a json column in Athena using Presto stored procedure

The Scenario
I've chose S3 folder location to create the table from a csv file, which has 1 column in JSON format. This needs to be exploded in way that creates many entries for one particular user & event.
The Problem
Athena Table looks something as follows:
agenda_data, event_id, partner_id, record_last_updated, user_id
"{'enclosed_data': {'task_active': 'true', 'status': 'completed'}, 'Agenda-1': {'currentProgress': '', 'timelines': '30/4/2020'}, 'Agenda-2': {'currentProgress': ' ', 'timelines': '25/4/2020'}, 'Agenda-3': {'currentProgress': ' ', 'timelines': '25/4/2020'}, 'Agenda-4': {'currentProgress': ' ', 'timelines': '28/4/2020'}, 'meta': {'foo': 'bar'}, 'Summary': {'finYear': '2020'}}, 'event_id': '20200407181839', 'record_last_updated': '2020-04-07T18:24:44.557362Z','user_id': '121000'}",20200407181839,Actionable,2020-04-06T13:20:31.114397Z,121000
"{'enclosed_data': {'consolidator': {'task_active': 'true', 'status': 'completed'},'Agenda-1': {'currentProgress': '', 'timelines': '25/4/2020'},'Agenda-2': {'currentProgress': 'On Going', 'timelines': '20/4/2020'},'Agenda-3': {'currentProgress': 'Completed', 'timelines': '07/4/2020'},'Agenda-4': {'currentProgress': ' ', 'timelines': '13/4/2020'},'meta': {'foo': 'bar'}, 'Summary': {'finYear': '2020'}}, event_id': '20200407202551',record_last_updated': '2020-04-07T20:32:48.215545Z', user_id': '12354'}",20200407202551,Actionable,2020-04-07T20:32:48.215545Z,12354
The Column agenda_data contains JSON data, which needs to be exploded. To put it clearly I'll repost the minimized structure of JSON.
{
"enclosed_data": {
"task_active": "true",
"status": "completed"
},
"Agenda-1": {
"currentProgress": "",
"timelines": "25/4/2020"
},
"Agenda-2": {
"currentProgress": "On Going",
"timelines": "20/4/2020"
},
"meta": {
"foo": "bar"
},
"Summary": {
"finYear": "2020"
}
},
"event_id": "20200407202551",
"record_last_updated": "2020-04-07T20:32:48.215545Z",
"user_id": "121000"
}
I need to project the Data of Agendas only when exploded, for same I tried resolving multiple blogs, I found Documents sensible though, here they go:
link1: Which helps very little
link2: Doesn't apply since I don't have Arrays in here
link3: Couldn't get either
The Expected output
The expected output is as follows:
event_id, partner_id, record_last_updated, user_id, agenda, currentProgress, timelines
20200407181839, Actionable, 2020-04-07T20:32:48.215545Z, 121000, Agenda-1, " ", "30/4/2020"
20200407181839, Actionable, 2020-04-07T20:32:48.215545Z, 121000, Agenda-2, " ", "25/4/2020"
20200407181839, Actionable, 2020-04-07T20:32:48.215545Z, 121000, Agenda-3, " ", "25/4/2020"
20200407181839, Actionable, 2020-04-07T20:32:48.215545Z, 121000, Agenda-4, " ", "28/4/2020"
20200407202551, Actionable, 2020-04-07T20:32:48.215545Z, 12354, Agenda-1, " ", "25/4/2020"
20200407202551, Actionable, 2020-04-07T20:32:48.215545Z, 12354, Agenda-2, "On Going", "20/4/2020"
20200407202551, Actionable, 2020-04-07T20:32:48.215545Z, 12354, Agenda-3, "Completed", "07/4/2020"
20200407202551, Actionable, 2020-04-07T20:32:48.215545Z, 12354, Agenda-4, " ", "13/4/2020"
EDIT #1
Success so far, that I could manage to parse json using presto function as follows:
QUERY
with meeting_data AS
(SELECT '{
"enclosed_data": {
"task_active": "true",
"status": "completed"
},
"Agenda-1": {
"currentProgress": "",
"timelines": "25/4/2020"
},
"Agenda-2": {
"currentProgress": "On Going",
"timelines": "20/4/2020"
},
"meta": {
"foo": "bar"
},
"Summary": {
"finYear": "2020"
}
},
"event_id": "20200407202551",
"record_last_updated": "2020-04-07T20:32:48.215545Z",
"user_id": "121000"
}' AS blob)
SELECT json_extract(blob,
'$["Agenda-1"]') AS agenda1, json_extract(blob, '$.enclosed_data.status') AS m_status, json_extract(blob, '$.Summary.finYear') AS finYear
FROM meeting_data
OUTPUT
agenda1, m_status, finYear
{"Agenda-1": {"currentProgress": "", "timelines":"25/4/2020"}},"completed", "2020-21"
OPEN QUESTIONS
I understood I can access the JSON when put it manually, I need this to be fetched from column one by one using loop, but how?
Once looped, how do I explode and get the expected output by repeating the other column values which aren't in JSON format?
Can this be achieved by writing a function/stored procedure in presto?

I want to get data in single array from different tables in Laravel 5.7

I am new in laravel and working on a project where I need to get the data from different tables in Laravel 5.7
Suppose I have 3 tables:
Main Table from which I need to fetch the data
Secondary Table 1
Secondary Table 2
Main Table Columns
id (auto increment primary key)
task_name (I have stored secondary table name here)
tid (task id)
assigned_to
description
Here is my code
public function viewTasks(){
$task_logs = TaskLog::orderBy('id','desc')->get();
foreach($task_logs as $task_log)
{
$table_name = $task_log['task_name'];
if(Schema::hasTable($table_name))
{
$tasks[] = DB::table($table_name)->where('id', $task_log->tid)->first();
}
}
return $tasks;
And here is the output:
[
{
"id": 220,
"uId": 324,
"document_name": "Photo Id",
"document_image": "image1.jpg",
"created_at": "2018-12-30 09:56:24",
"updated_at": "2018-12-30 09:56:24",
"status": 1,
},
{
"id": 114,
"uId": 382,
"makeModel": "Motorola 501",
"PhoneTitle": "New launched",
"price": "500",
"dealerName": "",
"created_at": "2018-12-30 09:56:24",
"updated_at": "2018-12-30 09:56:24",
"status": 1,
}
]
Output what I need:
[
{
"id": 220,
"uId": 324,
"document_name": "Photo Id",
"document_image": "image1.jpg",
"created_at": "2018-12-30 09:56:24",
"updated_at": "2018-12-30 09:56:24",
"status": 1,
"task_name": "documents",
"assigned to": 3,
"Description": "Description here",
},
{
"id": 114,
"uId": 382,
"makeModel": "Motorola 501",
"PhoneTitle": "New launched",
"price": "500",
"dealerName": "",
"created_at": "2018-12-30 09:56:24",
"updated_at": "2018-12-30 09:56:24",
"status": 1,
"task_name": "wishlists",
"assigned to": 2,
"Description": "Description here"
}
]
I have tried different ways using array_push function and array_merge etc to merge two arrays in a single array but no one worked. I don't know how can I implement this.
Please let me know if any information is missing in this question to understand and answering the question. Any help would be greatly appreciated. Thanks in advance.
you can merge different objects in PHP, In this case you have to use put the variables in the single array in foreach and you will get the required format of data.
public function viewTasks(){
$array = [];
$task_logs = TaskLog::orderBy('id','desc')->get();
foreach($task_logs as $task_log)
{
$table_name = $task_log['task_name'];
if(Schema::hasTable($table_name))
{
$tasks[] = DB::table($table_name)->where('id', $task_log->tid)->get();
$array[] = array_merge($tasks,$task_log);
}
}
return $array;
Can you try this... hope it work
public function viewTasks(){
$i = 0;
$task_logs = TaskLog::orderBy('id','desc')->get();
foreach($task_logs as $task_log)
{
$table_name = $task_log['task_name'];
if(Schema::hasTable($table_name)){
$tasks[$i] = DB::table($table_name)->where('id', $task_log->tid)->first();
$tasks[$i]['task_name'] = $task_log['task_name'];
$tasks[$i]['assigned_to'] = $task_log['assigned_to'];
$tasks[$i]['description'] = $task_log['description'];
$i++;
}
}
return $tasks;

How can I assert that data represented by two JSON strings is equal?

For the first time I'm trying to use unit tests for my projects, but I'm blocked with a test asserting that a model is properly stored.
This is the API controller I want to test:
public function store(QuestionFormRequest $request)
{
$questionRequest = $request->question();
$question = new Question($questionRequest);
$question->save();
$question->answers()->createMany($questionRequest['answers']);
return response()->json($question->load('answers'), 201);
}
This is my test:
public function it_can_store_a_question()
{
$surveyFactory = factory(Survey::class)->create();
$themeFactory = factory(Theme::class)->create();
$pillarFactory = factory(Pillar::class)->create();
$questionToStore = [
'survey_id' => $surveyFactory->id,
'theme_id' => $themeFactory->id,
'pillar_id' => $pillarFactory->id,
'name' => 'question',
'type' => 'simple',
'answers' => [
[
'label' => 'reponse1',
'points' => '3',
],
[
'label' => 'reponse2',
'points' => '5',
]
]
];
$response = $this->post('/api/1.0/question', $questionToStore);
$response->assertStatus(201);
$expectedQuestion = Question::with('answers')->get()->first();
$this->assertEquals(json_encode($expectedQuestion), $response->getContent());
}
This is the result:
Failed asserting that two strings are equal.
Expected :'{"id":1,"survey_id":1,"theme_id":1,"pillar_id":1,"name":"question","type":"simple","created_at":"2017-08-29 08:54:45","updated_at":"2017-08-29 08:54:45","deleted_at":null,"answers":[{"id":1,"question_id":1,"label":"reponse1","points":3,"description":"","created_at":"2017-08-29 08:54:45","updated_at":"2017-08-29 08:54:45","deleted_at":null},{"id":2,"question_id":1,"label":"reponse2","points":5,"description":"","created_at":"2017-08-29 08:54:45","updated_at":"2017-08-29 08:54:45","deleted_at":null}]}'
Actual :'{"survey_id":1,"theme_id":1,"pillar_id":1,"name":"question","type":"simple","updated_at":"2017-08-29 08:54:45","created_at":"2017-08-29 08:54:45","id":1,"answers":[{"id":1,"question_id":1,"label":"reponse1","points":3,"description":"","created_at":"2017-08-29 08:54:45","updated_at":"2017-08-29 08:54:45","deleted_at":null},{"id":2,"question_id":1,"label":"reponse2","points":5,"description":"","created_at":"2017-08-29 08:54:45","updated_at":"2017-08-29 08:54:45","deleted_at":null}]}'
In fact, the result is right. But not in the same order.
What am I doing wrong in my test?
Thanks.
For your example you could use assertions shipped with phpunit/phpunit:
assertJsonFileEqualsJsonFile()
assertJsonStringEqualsJsonString()
assertJsonStringEqualsJsonFile()
or
assertEquals()
assertJsonFileEqualsJsonFile()
However, when I - for the sake of an example - extract the two JSON strings into separate files
expected.json
actual.json
and then format the JSON in PhpStorm with option + command + L and run the following test
use PHPUnit\Framework;
final class JsonTest extends Framework\TestCase
{
public function testJsonEquals()
{
$expected = __DIR__ . '/expected.json';
$actual = __DIR__ . '/actual.json';
$this->assertJsonFileEqualsJsonFile($expected, $actual);
}
}
the test fails with
Failed asserting that '{\n
"id": 1,\n
"survey_id": 1,\n
"theme_id": 1,\n
"pillar_id": 1,\n
"name": "question",\n
"type": "simple",\n
"created_at": "2017-08-29 08:54:45",\n
"updated_at": "2017-08-29 08:54:45",\n
"deleted_at": null,\n
"answers": [\n
{\n
"id": 1,\n
"question_id": 1,\n
"label": "reponse1",\n
"points": 3,\n
"description": "",\n
"created_at": "2017-08-29 08:54:45",\n
"updated_at": "2017-08-29 08:54:45",\n
"deleted_at": null\n
},\n
{\n
"id": 2,\n
"question_id": 1,\n
"label": "reponse2",\n
"points": 5,\n
"description": "",\n
"created_at": "2017-08-29 08:54:45",\n
"updated_at": "2017-08-29 08:54:45",\n
"deleted_at": null\n
}\n
]\n
}\n
' matches JSON string "{
"survey_id": 1,
"theme_id": 1,
"pillar_id": 1,
"name": "question",
"type": "simple",
"updated_at": "2017-08-29 08:54:45",
"created_at": "2017-08-29 08:54:45",
"id": 1,
"answers": [
{
"id": 1,
"question_id": 1,
"label": "reponse1",
"points": 3,
"description": "",
"created_at": "2017-08-29 08:54:45",
"updated_at": "2017-08-29 08:54:45",
"deleted_at": null
},
{
"id": 2,
"question_id": 1,
"label": "reponse2",
"points": 5,
"description": "",
"created_at": "2017-08-29 08:54:45",
"updated_at": "2017-08-29 08:54:45",
"deleted_at": null
}
]
}
".
--- Expected
+++ Actual
## ##
{
- "id": 1,
"survey_id": 1,
"theme_id": 1,
"pillar_id": 1,
"name": "question",
"type": "simple",
+ "updated_at": "2017-08-29 08:54:45",
"created_at": "2017-08-29 08:54:45",
- "updated_at": "2017-08-29 08:54:45",
- "deleted_at": null,
+ "id": 1,
What you can see is that in fact the two JSON strings aren't equal, at least the order of the fields isn't the same. The problem with the output of the assertion is that it's not really helpful, simply because the assertion just compares the two strings.
assertEquals()
Nonetheless, when we json_decode() the content of the files and then assert using assertEquals() like this:
use PHPUnit\Framework;
final class JsonTest extends Framework\TestCase
{
public function testJsonEquals()
{
$expected = json_decode(file_get_contents(__DIR__ . '/expected.json'), true);
$actual = json_decode(file_get_contents(__DIR__ . '/actual.json'), true);
$this->assertEquals($expected, $actual);
}
}
then the test fails again, but with output that is actually much more helpful:
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
## ##
- 'deleted_at' => null
As you can see, there's an actual difference between the data - the root object node is missing a deleted_at field with a value of null.
Note So, the question boils down to
do you want to assert that two JSON strings are equal or
do you want to assert that the data represented by two JSON strings are equal
Depending on which is important to you, pick whichever assertion makes more sense to you.
The JSON output is just a little bit different - instead of comparing the actual string output you'd be better to use one of Laravel's JSON assertions to ensure the output contains what you expect. Take a look at the available assertions in the documentation. You might use $response->assertJson() or $response->assertJsonFragment() to ensure the required items are returned.

How to Get JSON values Python

Learning Days
Code to the get the data in JSON Format
#...
cursor.execute("SELECT * FROM user")
response = {
"version": "5.2",
"user_type": "online",
"user": list(cursor),
}
response = json.dumps(response, sort_keys=False, indent=4, separators=(',', ': '))
print(response)
# ...
This produces output as
{
"version": "5.2",
"user_type": "online",
"user":
[
{
"name": "John",
"id": 50
},
{
"name": "Mark",
"id": 57
}
]
}
print(response["user"]) - TypeError: string indices must be integers
How do i access the values in JSON
json.dumps return a string, need a small conversion something like this, not sure is this the exact method to do
Solution:
response = JSONEncoder().encode(response )
response = JSONDecoder().decode(response )
response = json.loads(response )
print(response['user'[0]['id'])

Parse Json response and check missing key in Groovy

I want to read the values from below sample Json response
"items": [
{
"kind": "books#volume",
"id": "nwy8akU-nJUC",
"volumeInfo": {
"title": "Light In August",
"authors": ["William Faulkner"],
"publisher": "Random House",
"publishedDate": "2013-07-05",
"pageCount": 384,
"categories": ["Fiction"]
},
"saleInfo": {
"country": "IN",
"saleability": "FOR_SALE",
"isEbook": true,
"listPrice": {
"amount": 339.0,
"currencyCode": "INR"
}
}
}
]
When I read the response and iterate in a loop as an array, amount & currencyCode keys are missing in the second result thus its totally ignored in the array. Please help me as to how to loop through the key and identify if the key is not there then set value null.
I have written a below sample code
testRunner.runTestStepByName("Volumes1")
def responseContent = context.testCase.getTestStepByName("Volumes1").getProperty("response").value
JsonSlurper jsonResponseContent = new JsonSlurper()
def jsonResponseObject = jsonResponseContent.parseText(responseText)
titleCount = jsonResponseObject.items.volumeInfo.title.size
arrayTitle = jsonResponseObject.items.volumeInfo.title
arrayAuthor = jsonResponseObject.items.volumeInfo.authors
arrayPublisher = jsonResponseObject.items.volumeInfo.publisher
arrayPublishDate = jsonResponseObject.items.volumeInfo.publishedDate
arrayPageCount = jsonResponseObject.items.volumeInfo.pageCount
arrayCategories = jsonResponseObject.items.volumeInfo.categories
arrayAmount = jsonResponseObject.items.saleInfo.listPrice.amount
arrayCurrencyCode = jsonResponseObject.items.saleInfo.listPrice.currencyCode
Try:
arrayAmount = jsonResponseObject.items.saleInfo?.listPrice?.amount
arrayCurrencyCode = jsonResponseObject.items.saleInfo?.listPrice?.currencyCode
println "list price amount : ${arrayAmount}"
println "list price currency code : ${arrayCurrencyCode}"
The above uses null safe. Refer here for more details.