I'm trying to update a JSON field in PostgreSQL (v14) using PowerShell system.data.dataset method but keep getting error:
"ERROR [42883] ERROR: operator does not exist: json = unknown;
I just want to read the JSON field as a string then update a part of it (I don't care of identifying JSON fields I just want to treat it and update it as a string).
$DBConnectionString = "Driver={PostgreSQL UNICODE(x64)};Server=$Server;Port=$Port;Database=$DB;Uid=$Uid;Pwd=$Pass;"
$Con = New-Object System.Data.Odbc.OdbcConnection;
$Con.ConnectionString = $DBConnectionString;
$Con.Open();
$Query = "Select ID, INFO from TestTable"
$ODBCDataAdapter = New-Object system.Data.odbc.odbcDataAdapter
$DataSet = New-Object System.Data.DataSet
$ODBCCommandSelect = New-Object System.Data.Odbc.OdbcCommand($query,$Con)
$ODBCDataAdapter.SelectCommand = $ODBCCommandSelect #execute Query
#setup the command to execute to fill the in-memory table
$ODBCDataAdapter.SelectCommand = $ODBCCommandSelect #execute Query
$ODBCDataAdapter.Fill($dataSet) #in memory database table created
$Con.Close();
#I now have a $dataset containing the rows from the table
For($i=0; $i -le ($dataset.tables[0].rows.Count -1); $i++){
IF($dataset.Tables[0].rows[$i].Info -match "Josh"){
$dataSet.Tables[0].Rows[$i].Info = $dataSet.Tables[0].Rows[$i].Info -replace "Josh", "Barbara"
}
$ODBCCommandUpdate = New-Object System.Data.Odbc.OdbcCommandBuilder($ODBCDataAdapter)
$ODBCDataAdapter.UpdateCommand = $ODBCCommandUpdate.GetUpdateCommand()
#all working fine until here:
$ODBCDataAdapter.Update($Dataset.tables[0])
$ODBCDataAdapter.Dispose()
$Dataset.Dispose()
$con.Close()
$con.Dispose()
This works for all the other fields (i.e. non JSON).
I tried a select statement like this:
$Query = "Select ID, CAST(INFO as Varchar) from TestTable"
And although it didn't error out, it still didn't update the field.
here's what the table looks like
ID INFO(JSON)
1 { "customer": "John Doe", "items": {"product": "Beer","qty": 6}}
2 { "customer": "Lily Bush", "items": {"product": "Diaper","qty": 24}}
3 {"customer": "Josh William", "items": {"product": "Toy Car", "qty": 1}}
4 { "customer": "Mary Clark", "items": {"product": "Toy Train","qty": 2}}
Help me Stackoverflowbi Wan Kenobi, you're my only hope.
Found the answer:
I added a line that changes the way that CommandBuilder creates the update statement:
$ODBCCommandUpdate.ConflictOption =3
$now the builder changes the Where clause in the update statement to only include the primary key.
Related
I am a newbie to Powershell and I couldn't find this on googling but how do I build a json structure with an array inside? Is it through a custom object? Have been a bit confused by the syntax that I have seen online and not sure what is the recommended way to do it. Eventually I need to be able to save it to a JSON file too.
json_ret = {
"a": 4,
"b": [ {"c" : 5, "d": "text", "e": "foo"}]
}
Yes you can build a json object through a PSCustomObject:
[PSCustomObject]#{
a = 4
b = #([ordered]#{
c = 5
d = "text"
e = "foo"
})
} | ConvertTo-Json
First we create PSObject using its type accelerator PSCustomObject.
Then we define the root key and value "a", and we have to create an array inside "b".
The #() statement creates an array, but we can't we create key-value pairs in array. So we use #{} to create hashtable. Before it [ordered] flag says the hashtable to keep the exact structure as we have created it.
Then we define the array values, and after that close the internal array-hashtable.
Now we end the PSCustomObject and pipe it ConvertTo-Json. Now you get a converted json.
Footnotes
If you want to dump the json to a file, then use this:
[PSCustomObject]#{
a = 4
b = #([ordered]#{
c = 5
d = "text"
e = "foo"
})
} | ConvertTo-Json | Out-File "Filepath"
If you want to save json to a variable:
$variable = ([PSCustomObject]#{
a = 4
b = #([ordered]#{
c = 5
d = "text"
e = "foo"
})
} | ConvertTo-Json)
If you want to create the JSON document directly, as a string, it's simplest to use a verbatim here-string:
$json_ret = #'
{
"a": 4,
"b": [ {"c" : 5, "d": "text", "e": "foo"}]
}
'#
You can easily save that to a file with $json_ret | Set-Content file.json, for instance.
By contrast, if you want to construct your data as an object graph first, to be converted to JSON with ConvertTo-Json later, see Wasif_Hasan's helpful answer.
As for what you tried:
An unquoted { ... } construct is a script block, which is a piece of PowerShell code for later invocation on demand - and the contents of your JSON document happen not to constitute valid PowerShell code, causing construction of the script block to fail.
If using variables, then can create body including array like shown below, where $text is the variable. No need to user ConvertTo-Json and can easily copy the body from postman directly.
$text = "ABC"
# Post Body
$body = #"
{
"name" = "$text",
"description" = "$text",
"myArray": [
{
"id": "2c91808680d3c34b0180dc81d78c21e9",
"type": "myType",
"name": "myName"
}
]
}
"#
$body
I have imported some JSON data and converted it to a PowerShell Object. I would like to understand how to retrieve specific portions of said data.
test.json:
{
"Table": {
"Users": {
"Columns": [ "[Id]",
"[FName]",
"[MName]",
"[SName]",
"[UName]",
"[Pasword]" ],
"data": "CustomUserData"
},
"Roles": {
"Columns": [ "[Id]",
"[Role]",
"[Description]" ],
"data": "CustomRoleData"
}
}
}
Import to PS Object:
$userdata = Get-Content .\test.json |ConvertFrom-Json
Retrieve and format column data:
PS> $userdata = Get-Content ./test.json |ConvertFrom-Json
PS> $columns = $userdata.Table.Users.Columns -join ","
PS> $columns
[Id],[FName],[MName],[SName],[UName],[Pasword]
Example retrieval of custom data:
PS> $userdata.Table.Users.data
CustomUserData
What I would like to do is:
Select just the table names. When I try and do this by calling $userdata.table I get the following:
PS> $userdata.Table |Format-List
Users : #{Columns=System.Object[]; data=CustomUserData}
Roles : #{Columns=System.Object[]; data=CustomRoleData}
What I am looking for is just a list of the table names, in this case - Users,Roles
I would also like to know how to leverage this to create a ForEach loop which cycles through each table name and prints the columns associated with each table - ultimately I will be using this to craft a SQL query.
Thank you!
Maybe this can help you.
It is a small function to output the property names recursively.
function Get-Properties($obj, [int]$level = 0) {
$spacer = " "*$level
$obj.PSObject.Properties | ForEach-Object {
$spacer + $_.Name
if ($_.Value -is [PSCustomObject]){
Get-Properties $_.Value ($level + 2)
}
}
}
In your case, you can use it like this:
$userdata = Get-Content ./test.json | ConvertFrom-Json
Get-Properties $userData
The console output will look like this:
Table
Users
Columns
data
Roles
Columns
data
You should read this if: you are trying to find out how to transform SQL server data into JSON and put it into a text .json file
Question:
Can someone tell me what's wrong with this code? My goal is to read data from a SQL Server table, convert it to JSON and then save the result as a JSON text file. The code runs but the resulting .json file just has:
{
"FieldCount": 11
},
{
repeated over and over again and nothing more.
My code:
$instance = "localhost\SQLEXPRESS"
$connectionString = "Server=$Instance; Database=myDB;Integrated Security=True;"
$query = "Select * from myTable"
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$connection.Open()
$command = $connection.CreateCommand()
$command.CommandText = $query
$result = $command.ExecuteReader()
$result | ConvertTo-Json | Out-File "file.json"
$connection.Close()
Update:
Will award the answer to postanote as technically he/she answered my original question (although I will caveat and say I have not tried it).
However I would recommend either Mike's answer or what I eventually ended up going with, using BCP:
bcp "select * from myTable FOR JSON AUTO" queryout "C:\filepath\testsml.json" -c -S ".\SQLEXPRESS" -d myDBName -T
Note that the JSON AUTO will automatically come up with a json scehma for you vs. JSON pipeline which allows you to customize it.
You have to install BCP first:
https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-2017
If you are using sql server express 2016 or later you should be able to do it on the database side using FOR JSON clause. Try something like
$instance = "localhost\SQLEXPRESS"
$connectionString = "Server=$Instance; Database=myDB;Integrated Security=True;"
$query = "Select * from myTable FOR JSON AUTO"
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$connection.Open()
$command = $connection.CreateCommand()
$command.CommandText = $query
$command.ExecuteScalar() | Out-File "file.json"
Try something like this …
### Exporting SQL Server table to JSON
Clear-Host
#--Establishing connection to SQL Server --#
$InstanceName = "."
$connectionString = "Server=$InstanceName;Database=msdb;Integrated Security=True;"
#--Main Query --#
$query = "SELECT * FROM sysjobs"
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$connection.Open()
$command = $connection.CreateCommand()
$command.CommandText = $query
$result = $command.ExecuteReader()
$table = new-object "System.Data.DataTable"
$table.Load($result)
#--Exporting data to the screen --#
$table | select $table.Columns.ColumnName | ConvertTo-Json
$connection.Close()
# Results
{
"job_id": "5126aca3-1003-481c-ab36-60b45a7ee757",
"originating_server_id": 0,
"name": "syspolicy_purge_history",
"enabled": 1,
"description": "No description available.",
"start_step_id": 1,
"category_id": 0,
"owner_sid": [
1
],
"notify_level_eventlog": 0,
"notify_level_email": 0,
"notify_level_netsend": 0,
"notify_level_page": 0,
"notify_email_operator_id": 0,
"notify_netsend_operator_id": 0,
"notify_page_operator_id": 0,
"delete_level": 0,
"date_created": "\/Date(1542859767703)\/",
"date_modified": "\/Date(1542859767870)\/",
"version_number": 5
}
The "rub" here is that the SQL command FOR JSON AUTO even with execute scalar, will truncate JSON output, and outputting to a variable with VARCHAR(max) will still truncate. Using SQL 2016 LocalDB bundled with Visual Studio if that matters.
Here is a small Powershell code snippet:
$users = New-Object System.Collections.ArrayList
$userAsJson = '
{
"name" : "abc",
"companies" : ["facebook", "google"]
}'
$user = $userAsJson | ConvertFrom-Json
$null = $users.Add($user)
$users | ConvertTo-Json -Depth 5
It gives me the following expected output:
{
"name": "abc",
"companies": [
"facebook",
"google"
]
}
Now, I'm dynamically trying to create the companies list. I tried all possible things which I can think of. Here is what I have tried:
$company = New-Object System.Collections.ArrayList
$null = $company.Add('facebook')
$null = $company.Add('google')
$b = $company.ToArray()
$users = New-Object System.Collections.ArrayList
$userAsJson = '
{
"name" : "abc",
"companies" : $b
}'
$user = $userAsJson | ConvertFrom-Json
$null = $users.Add($user)
$users | ConvertTo-Json -Depth 5
Can anyone suggest me what is the best way to achieve it?
PowerShell's strength is in staying in the realm objects, until the time comes to interface with the outside world, such as when writing to a file or creating a string representation of these objects.
In your case that means:
# Array of companies; statically constructed here, but creating it
# dynamically works just as well.
$company = (
'facebook',
'google'
)
# Initialize the output collection.
# Note: Creating a [System.Collections.ArrayList] instance is
# advisable for building up *large* arrays *incrementally*.
# For smallish arrays, using regular PowerShell arrays will do; e.g.:
# $users = #() # initialize array
# $users += ... # append to array, but be aware that a *new* array
# is created behind the scenes every time.
$users = New-Object System.Collections.ArrayList
# Add a user based on the $company array defined above as
# a [pscustomobject]
$null = $users.Add(
[pscustomobject] #{
name = 'abc'
companies = $company
}
)
# After all users have been added *as objects*, convert them to JSON.
$users | ConvertTo-Json -Depth 5
The above yields (based on a single object having been added; with more, a JSON array would be output):
{
"name": "abc",
"companies": [
"facebook",
"google"
]
}
The following JSON is getting returned from OData API service:
{
"d": {
"results": [
{
"FileSystemObjectType": 0,
"Id": 1,
"ContentTypeId": "0x0100BC97B2F575CB0C42B79549F3BABD32A8",
"Title": "Nokia California",
"Address": "200 South Matilda Avenue\nW Washington Ave\n94086 Sunnyvale, California\nUnited States of America",
"ID": 1,
"Modified": "2014-02-24T10:06:39Z",
"Created": "2014-02-24T10:06:39Z",
"AuthorId": 12,
"EditorId": 12,
"OData__UIVersionString": "1.0",
"Attachments": false,
"GUID": "d12aafad-502a-4968-a69e-36a7ea05ec80"
}
]
}
}
and saved as a string into variable named $data
An attempt to convert a JSON-formatted string to a custom object using ConvertFrom-Json cmdlet:
$results = $data | ConvertFrom-Json
gives the following error:
ConvertFrom-Json : Cannot convert the JSON string because a dictionary
that was converted from the string contains the duplicated keys 'Id'
and 'ID'.
Is there any way to convert the specified JSON-formatted string in PowerShell?
This is how I have done with it:
$results = $data.ToString().Replace("ID", "_ID") | ConvertFrom-Json
Note, both examples assume the JSON is stored in the $jsonstring variable.
In PowerShell Core, ConvertFrom-Json -AsHashtable is the easiest alternative:
$json = $jsonstring | ConvertFrom-Json -AsHashtable
$json['d']['results']
Name Value
---- -----
Modified 2/24/2014 10:06:39 AM
Title Nokia California
Attachments False
ID 1
ContentTypeId 0x0100BC97B2F575CB0C42B79549F3BABD32A8
GUID d12aafad-502a-4968-a69e-36a7ea05ec80
Created 2/24/2014 10:06:39 AM
EditorId 12
AuthorId 12
Address 200 South Matilda Avenue…
Id 1
OData__UIVersionString 1.0
FileSystemObjectType 0
In Windows PowerShell, you can use the Deserialize(String, Type) method from the JavaScriptSerializer Class.
Add-Type -AssemblyName System.Web.Extensions
$serializer = [Web.Script.Serialization.JavaScriptSerializer]::new()
$json = $serializer.Deserialize($jsonstring, [hashtable])
$json['d']['results']
In PowerShell V1.0, or in PowerShell V2.0 when the JSON is too big, I still use a convertion to XML :
Add-Type -AssemblyName System.ServiceModel.Web, System.Runtime.Serialization
function Convert-JsonToXml
{
PARAM([Parameter(ValueFromPipeline=$true)][string[]]$json)
BEGIN
{
$mStream = New-Object System.IO.MemoryStream
}
PROCESS
{
$json | Write-String -stream $mStream
}
END
{
$mStream.Position = 0
try
{
$jsonReader = [System.Runtime.Serialization.Json.JsonReaderWriterFactory]::CreateJsonReader($mStream,[System.Xml.XmlDictionaryReaderQuotas]::Max)
$xml = New-Object Xml.XmlDocument
$xml.Load($jsonReader)
$xml
}
finally
{
$jsonReader.Close()
$mStream.Dispose()
}
}
}
Using this code you can loop thru your items you can test :
$a = Get-Content C:\temp\jsontest.txt
$b.root.d.results.Item
$b.root.d.results.Item[7].Id[0].InnerText
(Edited)
In you case I would only replace the expected duplicate ID/Id
$data = Get-Content C:\temp\jsontest.txt -Raw
$datacorrected = $a -creplace '"Id":','"Id-minus":'
$psJsonIn = $datacorrected | ConvertFrom-Json
If really you've got unexpected duplicate you can write a function that trap the convertion error due to duplicated key and replace one.
ConvertFrom-JSON it going to try to create a PS Custom Object from the JSON string. PowerShell object property names are case-insensitive, so "ID" and "id" represent the same property name. You're going to have to do something with those duplicate property names in your JSON before you try to do that conversion, or it's going to fail.
I used the ToLower() before converting the json to object, resolved my issue.
$sdf = $data.ToLower() | ConvertFrom-Json