Bitbucket API Invoke-RestMethod, Powershell, Get data to Json file,webscraping - json

I try to do exact same thing that Bibucket API explained
BitBucket says/explain
https://confluence.atlassian.com/bitbucket/repository-resource-423626331.html
GET https://api.bitbucket.org/2.0/repositories/{owner}/{repo_slug}
{
"scm": "hg",
"has_wiki": true,
"description": "Site for tutorial101 files",
"links": {
"watchers": {
"href": "https://api.bitbucket.org/2.0/repositories/tutorials/tutorials.bitbucket.org/watchers"
},
"commits": {
"href": "https://api.bitbucket.org/2.0/repositories/tutorials/tutorials.bitbucket.org/commits"
},
"self": {
"href": "https://api.bitbucket.org/2.0/repositories/tutorials/tutorials.bitbucket.org"
},
"html": {
"href": "https://bitbucket.org/tutorials/tutorials.bitbucket.org"
},
"avatar": {
"href": "https://bitbucket-assetroot.s3.amazonaws.com/c/photos/2012/Nov/28/tutorials.bitbucket.org-logo-1456883302-9_avatar.png"
},
"forks": {
"href": "https://api.bitbucket.org/2.0/repositories/tutorials/tutorials.bitbucket.org/forks"
},
"clone": [{
"href": "https://bitbucket.org/tutorials/tutorials.bitbucket.org",
"name": "https"
}, {
"href": "ssh://hg#bitbucket.org/tutorials/tutorials.bitbucket.org",
"name": "ssh"
}],
"pullrequests": {
"href": "https://api.bitbucket.org/2.0/repositories/tutorials/tutorials.bitbucket.org/pullrequests"
}
},
"fork_policy": "allow_forks",
"name": "tutorials.bitbucket.org",
"language": "html/css",
"created_on": "2011-12-20T16:35:06.480042+00:00",
"full_name": "tutorials/tutorials.bitbucket.org",
"has_issues": true,
"owner": {
"username": "tutorials",
"display_name": "tutorials account",
"uuid": "{c788b2da-b7a2-404c-9e26-d3f077557007}",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/users/tutorials"
},
"html": {
"href": "https://bitbucket.org/tutorials"
},
"avatar": {
"href": "https://bitbucket-assetroot.s3.amazonaws.com/c/photos/2013/Nov/25/tutorials-avatar-1563784409-6_avatar.png"
}
}
},
"updated_on": "2014-11-03T02:24:08.409995+00:00",
"size": 76182262,
"is_private": false,
"uuid": "{9970a9b6-2d86-413f-8555-da8e1ac0e542}"
}
I would like to do exact same thing by using powershell.
So, I made a code.
$username = ""
$password = ""
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password)))
$response=Invoke-RestMethod -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri https://bitbucket.org/susco/azuretoquickbase -Method Get
$response
I am sure that I could log in bitbucket since I saw the content of " $responce".
then, like Bitbucket says, I added the code like this to echo out Json file .
$username = ""
$password = ""
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password)))
$response=Invoke-RestMethod -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri https://bitbucket.org/susco/azuretoquickbase -Method Get
$response
$json = $response.Content|ConvertFrom-Json
$json.data.summaries
but it is causing error
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At line:11 char:27
+ $json = $response.Content|ConvertFrom-Json
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
The point is I am sure that I could log in and get to the page bitbucket says, but it is not getting json file and null error.
I want exactly to echo out like Bitbuckets page, How can i do that?

Invoke-RestMethod automatically converts JSON results into objects. If you want just the raw result, use Invoke-WebRequest and get the content property.
Or in your case, since it seems you want the object, just use $json = Invoke-RestMethod ....

Related

updating values in JSON using PUT method

I have a nested JSON file. I want to update some values with PUT method. I can change some values, but the problem is in the nested values.
$Authorization = "Bearer API-KEY"
$update = [ordered]#{
name = 'ppp'
asset_tag = 'BBB'
custom_fields = #(
[ordered] #{
Mediensatz = #(
[ordered] #{
value = "B2T-XXX"
}
)
}
)
}
$json = $update | ConvertTo-Json -Depth 100
#Write-Host $json
$response = Invoke-RestMethod 'URL' -Method Put -Body $json -ContentType 'application/json' -Headers #{'Authorization' = $Authorization}
My JSON file look like this:
{
"total": 888,
"rows": [
{
"id": 11,
"name": "AAA",
"asset_tag": "CCC",
"model": {
"id": 34,
"name": "TTT"
},
"user_can_checkout": true,
"custom_fields": {
"Mediensatz": {
"field": "lll",
"value": "B2T-YYY",
"field_format": "ANY"
},
"Ueberschreibschutz": {
"field": "lll",
"value": "2019-07-10",
"field_format": "DATE"
}
}
}
I can change the "name" and "asset_tag", but the problem is "value" in custom_fields → Mediensatz → value.
When I try to run my code with Write-Host $json it's look like this:
{
"name": "PPP",
"asset_tag": "BBB",
"custom_fields": [
{
"Mediensatz": [
{
"value": "B2T-XXX"
}
]
}
]
}

Need to search a text file for a string and capture a second string located after the first

I'm working with an API call and I need to parse the JSON returned by the API in PowerShell. I'm searching for a particular name and then I need to capture a another item that follows it.
$json = #"
{
"Test": [
{
"association": "(None)",
"description": "",
"displayName": "BusIntel48.png",
"galleryImage": "[PlugIn]Images;Trebuchet.PlugIn.Images.Images.Common.BusIntelligence.BusIntel48.png",
"id": "944f73cc5c1a812eaa2ade43bd9ca0c9a40f963c56",
"links": [],
"localizedScopeName": "Built in",
"name": "BusIntel48.png",
"parentFolder": "ImagesCommonBusIntelligence",
"parentIsScopeFolder": false,
"scope": "FromResource",
"scopeOwner": "",
"standInKey": "DefType:ImageDef#Scope:FromResource#Id:944f73cc5c1a812eaa2ade43bd9ca0c9a40f963c56"
},
{
"association": "(None)",
"description": "",
"displayName": "BusProcActOnEvent16.png",
"galleryImage": "[PlugIn]Images;Trebuchet.PlugIn.Images.Images.Common.BusIntelligence.BusProcActOnEvent16.png",
"id": "944f73cc5cb33339be6db84987baef3f7190c931cf",
"links": [],
"localizedScopeName": "Built in",
"name": "BusProcActOnEvent16.png",
"parentFolder": "ImagesCommonBusIntelligence",
"parentIsScopeFolder": false,
"scope": "FromResource",
"scopeOwner": "",
"standInKey": "DefType:ImageDef#Scope:FromResource#Id:944f73cc5cb33339be6db84987baef3f7190c931cf"
}
]
}
"#
$x = $json | ConvertFrom-Json
$name = $x.Test | where { $_.name -eq "BusIntel48.png" }
I need to search for the name of the image file (ex. "name": "BusIntel48.png") and then capture the string after "standInKey: (ex. DefType:ImageDef#Scope:FromResource#Id:944f73cc5c1a812eaa2ade43bd9ca0c9a40f963c56).
Just need a change in last line.
You can also select one property by using :
$name = $x.Test | where { $_.name -eq "BusIntel48.png" } | Select-Object -Property "standInKey"

JSON data from GET method

How can I retrieve json data in .json file from API get query of product?
Example: https://api.abcd.com/v1/products/search.json?q=ball
header[key=abc, value=xyz]
From get query fetch json data from postman like
{
"Results": [
{
"Id": 5481192,
"Name": " Shirt",
"Description": " tweens. 100% Cotton.",
"ShortDescription": " Raglan Shirt",
"Number": "4253121",
"ImageUrl": "media/295428",
"VirtualSampleImages": [
{
"Id": 2245428,
"ImageUrl": "virtualsample/2529548"
}
],
"ConfigId": "23",
"Supplier": {
"Id": 613,
"Name": "a",
"AsiNumber": "3529721",
"Phone": {
"Work": "(451) 488-0417",
"$index": 1
}
},
"Price": {
"Quantity": 11,
"Price": 133.7,
"Cost": 61.85,
"DiscountCode": "P",
"CurrencyCode": "USD"
},
"IsNew": false,
"IsConfirmed": false,
"HasVirtualSample": true
}
],
"Selections": {},
"Dimensions": {},
"Links": {
"Self": "?q=shirts&page=1&rpp=1",
"Next": "?q=shirts&page=2&rpp=1"
},
"Query": "shirts",
"Breadcrumb": "\"shirts\"",
"Page": 1,
"ResultsPerPage": 21,
"ResultsTotal": 78,
"SuppliersTotal": 6677,
"CompletedIn": 7
}
I want to save the json data to .json file (automatically) after that to MySQL database with individual columns.
You can simply use curl for the save-to-file process.
curl 'https://api.abcd.com/v1/products/search.json?q=ball' -H 'Accept: application/json, text/plain, */*' -H 'key: abc' -H 'value: xyz' -o dump.json
Then, you can load this file into table with LOAD_FILE :
INSERT INTO table_name (STAMP,file_content) VALUES(NOW(),LOAD_FILE("dump.json"));
The whole process depends on you scripting/programming language. Can you tell us more about your technical context ?
$curl = curl_init();
$options=array(CURLOPT_URL=>"http://api.abc.com/v1/products/search.json?q=ball",CURLOPT_RETURNTRANSFER =>true,
CURLOPT_ENCODING =>"",CURLOPT_FOLLOWLOCATION =>true,CURLOPT_MAXREDIRS => 10,CURLOPT_TIMEOUT=>30,CURLOPT_HTTP_VERSION=>CURL_HTTP_VERSION_1_0,
CURLOPT_CUSTOMREQUEST => "GET", CURLOPT_POSTFIELDS=>"",
CURLOPT_HTTPHEADER=> array("authorization: AsiMemberAuth client_id=50041351&client_secret=55700485cc39f1",
"cache-control: no-cache"), CURLOPT_HEADER=> true);
curl_setopt_array($curl, $options);
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err)
{
echo "cURL Error #:" . $err;
}
else
{
echo $response;
}

ARM Template DSC: configuration does not 'see' protectedSettings.configurationArguments

I need to protect a DSC configuration parameter ([pscredential]RegistrationKey), so I have put it under "settings.protectedSettings.configurationData" thus:
"protectedSettings": {
"configurationArguments": {
"RegistrationKey": {
"UserName": "PLACEHOLDER_DONOTUSE",
"Password": "[parameters('dscAutomationRegistrationKey')]"
}
},
"configurationUrlSasToken": "[parameters('artifactsLocationSasToken')]"
}
I get the error:
"VM has reported a failure when processing extension 'Microsoft.Powershell.DSC'. Error message: \"The DSC Extension failed to execute: Mandatory
parameter RegistrationKey is missing.
If I move RegistrationKey out of "settings.protectedSettings.configurationArguments", into "settings.configurationArguments", it works, therefore, I assume there is nothing wrong with the syntax, so I believe it is to do with PsDscAllowPlainTextPassword = $true that wasn't included in the DSC configuration.
(I tried to include the configuration block in the PS1 file, but this threw an error, suggesting this can't be done)
I have now written a configurationdata .psd1 file, containing the following:
$ConfigData = #{
AllNodes = #(
#{
NodeName = "*"
PsDscAllowPlainTextPassword = $true
}
)
}
and referenced it in settings.configurationdata.url.
This now results in the same error as before: VM has reported a failure...
The ARM template is called from PowerShell:
$oAutomationAccount = Get-AzureRmAutomationAccount -ResourceGroupName $AAresourceGroupName -Name $AutomationAccountName
$RegistrationInfo = $oAutomationAccount | Get-AzureRmAutomationRegistrationInfo
$DscRegKeyString = $RegistrationInfo.PrimaryKey
$ssDscAutomationRegistrationKey = (ConvertTo-SecureString -string $DscRegKeyString -AsPlainText -Force)
#Automation Account EndPoint Uri
$DscRegistrationUrl = $RegistrationInfo.Endpoint
$params = #{
artifactsLocationSasToken = $TemplateSas
vmName = "XYZ"
dscAutomationRegistrationKey = $ssDscAutomationRegistrationKey
dscAutomationRegistrationUrl = $DscRegistrationUrl
dscNodeConfigurationName = "CreateAFolder.localhost"
dscTimeStamp = (Get-Date -f "MM/dd/yyyy H:mm:ss tt") #"MM/dd/yyyy H:mm:ss tt"
dscResourceUrl = $DscResourceUrl
dscConfigurationUrl = $DscConfigurationUrl
dscResourceScript = $DscResourceScriptName
dscResourceFunction = "ConfigureLCMforAAPull"
#sequenceId = $sequenceId
}
New-AzureRmResourceGroupDeployment #params `
-Name "$TemplateInstance-$branch" `
-ResourceGroupName $DeploymentResourceGroup.ResourceGroupName `
-Mode Incremental `
-DeploymentDebugLogLevel All `
-TemplateUri $TemplateUri `
-Verbose
Where I believe the parameters are being passed as the correct types.
What am I doing wrong?
Reference template: https://github.com/Azure/azure-quickstart-templates/blob/master/dsc-extension-azure-automation-pullserver/azuredeploy.json
Updated to use a newer DSC schema:https://blogs.msdn.microsoft.com/powershell/2016/02/26/arm-dsc-extension-settings/
this is the template I've been using for node onboarding:
{
"name": "xxx",
"type": "Microsoft.Compute/virtualMachines/extensions",
"location": "[parameters('location')]",
"apiVersion": "2015-06-15",
"dependsOn": [
"xxx"
],
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.22",
"autoUpgradeMinorVersion": false,
"protectedSettings": {
"Items": {
"registrationKeyPrivate": "[parameters('registrationData')]"
}
},
"settings": {
"ModulesUrl": "https://github.com/Azure/azure-quickstart-templates/raw/master/dsc-extension-azure-automation-pullserver/UpdateLCMforAAPull.zip",
"SasToken": "",
"ConfigurationFunction": "UpdateLCMforAAPull.ps1\\ConfigureLCMforAAPull",
"Properties": [
{
"Name": "RegistrationKey",
"Value": {
"UserName": "PLACEHOLDER_DONOTUSE",
"Password": "PrivateSettingsRef:registrationKeyPrivate"
},
"TypeName": "System.Management.Automation.PSCredential"
},
{
"Name": "RegistrationUrl",
"Value": "xxx",
"TypeName": "System.String"
},
{
"Name": "NodeConfigurationName",
"Value": "xxx",
"TypeName": "System.String"
},
{
"Name": "ConfigurationMode",
"Value": "ApplyAndMonitor",
"TypeName": "System.String"
},
{
"Name": "ConfigurationModeFrequencyMins",
"Value": 15,
"TypeName": "System.Int32"
},
{
"Name": "RefreshFrequencyMins",
"Value": 30,
"TypeName": "System.Int32"
},
{
"Name": "RebootNodeIfNeeded",
"Value": true,
"TypeName": "System.Boolean"
},
{
"Name": "ActionAfterReboot",
"Value": "ContinueConfiguration",
"TypeName": "System.String"
},
{
"Name": "AllowModuleOverwrite",
"Value": true,
"TypeName": "System.Boolean"
},
{
"Name": "Timestamp",
"Value": "MM/dd/yyyy H:mm:ss tt",
"TypeName": "System.String"
}
]
}
}
}
I know its using an old format, but that works so, meh.

Azure functions & Powershell: get response in plain JSON

Trying out some new features in Azure; Azure Functions running Powershell. One thing I'm struggling with for a while: I cannot get the output in plain JSON, instead in seems formatted and includes escape characters and is messed up double quoting.
function.json:
{
"bindings": [
{
"name": "req",
"type": "httpTrigger",
"direction": "in",
"webHookType": "genericJson",
"methods": [
"post",
"head",
"trace"
]
},
{
"name": "res",
"type": "http",
"direction": "out"
}
],
"disabled": false
}
run.ps1:
$ApiURI="http://avwx.rest/api/metar.php?station=ymml&format=JSON"
$Result = Invoke-RestMethod -Uri $ApiURI
Out-File -Encoding Ascii -FilePath $res -inputObject $Result
Results in following response:
"\r\n\r\nAltimeter : 1015\r\nCloud-List : {FEW 035}\r\nDewpoint : 11\r\nFlight-Rules : VFR\r\nOther-List : {}\r\nRaw-Report : YMML 210900Z 17010KT 9999 FEW035 19/11 Q1015 NOSIG\r\nRemarks : NOSIG\r\nRemarks-Info : \r\nRunway-Vis-List : {}\r\nStation : YMML\r\nTemperature : 19\r\nTime : 210900Z\r\nUnits : #{Altimeter=hPa; Altitude=ft; Temperature=C; Visibility=m; \r\n Wind-Speed=kt}\r\nVisibility : 9999\r\nWind-Direction : 170\r\nWind-Gust : \r\nWind-Speed : 10\r\nWind-Variable-Dir : {}\r\n\r\n\r\n\r\n"
But what I expect is something like:
{
"Altimeter": "1016",
"Cloud-List": [],
"Dewpoint": "12",
"Flight-Rules": "VFR",
"Other-List": [],
"Raw-Report": "YMML 210930Z 17011KT CAVOK 18/12 Q1016 NOSIG",
"Remarks": "NOSIG",
"Remarks-Info": {},
"Runway-Vis-List": [],
"Station": "YMML",
"Temperature": "18",
"Time": "210930Z",
"Units": {
"Altimeter": "hPa",
"Altitude": "ft",
"Temperature": "C",
"Visibility": "m",
"Wind-Speed": "kt"
},
"Visibility": "9999",
"Wind-Direction": "170",
"Wind-Gust": "",
"Wind-Speed": "11",
"Wind-Variable-Dir": []
}
What could be wrong here?
I'm not sure how you got that result, since Invoke-RestMethod store the data in the ResponseData property.
When you use Invoke-RestMethod it will convert the json data to a Powershell Object. In your run.ps1 it looks like you are going to save the PS Object to a file, not the JSON data.
If you are not going to modify the JSON data, it's really no use to convert it to a PSObject. So you could use Invoke-webrequest instead.
But in case you want to modify it.
$uri = "http://avwx.rest/api/metar.php?station=ymml&format=JSON"
$request = Invoke-RestMethod -Uri $uri
#Get the result and convert it back to JSON
$Result = $request.responseData | ConvertTo-Json
$Result | Out-File .\data.json -Encoding Ascii
#David - Yes, the following works for me in Azure Functions.
a. run.ps1 file content
$ApiURI="http://avwx.rest/api/metar.php?station=ymml&format=JSON"
$Result = Invoke-RestMethod -Uri $ApiURI | ConvertTo-Json
Out-File -Encoding Ascii -FilePath $res -inputObject $Result
b. Request result both in Azure Functions Portal and Postman.
{
"Altimeter": "1016",
"Cloud-List": [
[
"FEW",
"016"
]
],
"Dewpoint": "13",
"Flight-Rules": "VFR",
"Other-List": [],
"Raw-Report": "YMML 212100Z 28005KT 9999 FEW016 15/13 Q1016 NOSIG",
"Remarks": "NOSIG",
"Remarks-Info": {},
"Runway-Vis-List": [],
"Station": "YMML",
"Temperature": "15",
"Time": "212100Z",
"Units": {
"Altimeter": "hPa",
"Altitude": "ft",
"Temperature": "C",
"Visibility": "m",
"Wind-Speed": "kt"
},
"Visibility": "9999",
"Wind-Direction": "280",
"Wind-Gust": "",
"Wind-Speed": "05",
"Wind-Variable-Dir": []
}