Invalid key/value pair when adding json to vault - json

I tried: vault kv put -format=json secrets/path #file.json
Getting this error:
Failed to parse K=V data: invalid key/value pair "#file.json": json cannot unmarshal array into Go value of type map[string] interface {}
When trying vault kv put -format=json secrets/path file.json I get:
Failed to parse K=V data: invalid key/value pair "file.json": format must be key=value
Not sure what I'm doing wrong.

The first form is more correct if you're attempting to pass a JSON file as an argument. The second form is not referencing a file, it's just invalid syntax.
The error message on the first form suggests that the JSON file you have is formatted incorrectly. The required format will depend on your KV engine version. If you are working with a KV v2 engine, it is required to put your key:value pairs into a data top level map. If you are working with a KV v1 engine, each key:value pair needs to itself be a top level object.
KV v1:
{
"key": "value",
"foo": "bar",
"bar": "baz"
}
KV v2:
{
"data": {
"key": "value",
"foo": "bar",
"bar": "baz"
},
"options": {}
}
The -output-curl-string flag is great for inspecting what the vault CLI tool is doing under the hood, try adding it and see what transformations the binary applies to your commands.

Related

Terraform's external data source: STDOUT syntax unclear

I would like to use terraform's external data source to identify certain AWS EC2 instances:
data "external" "monitoring_instances" {
program = ["bash", "${path.module}/../bash/tf_datasource_monitoring.sh"]
query = {
env = var.env_stage
}
}
The bash script is using AWS CLI to return a list of instance IDs.
But I keep receiving this Error: command "bash" produced invalid JSON: json: cannot unmarshal array into Go value of type string
I don't understand what the expected syntax of my script's STDOUT would be for terraform to understand the result.
So let's assume the script is supposed to return 3 instance IDs i-1, i-2 and i-3.
What would be the correct JSON syntax to be returned to terraform?
Examples, that do NOT work:
{
"instances": [
"i-1",
"i-2",
"i-3"
]
}
[
"i-1",
"i-2",
"i-3"
]
It is a known issue in Terraform for provider-external: https://github.com/hashicorp/terraform-provider-external/issues/2. It was opened a while ago, unfortunately is still present for latest version (Terraform v1.011).
You may want to avoid returning JSON objects which contain arrays.
I had the same issue, while executing a python script to generate a dynamic bigquery schema, which is per definition an array of JSONs.
I solved it, by implementing a wrapper JSON with the schema as a string value (see dummy code below).
# get_dynamic_bigquery_schema.py
import json
bigquery_schema = [
{"name": "int_field", "type": "INTEGER", "mode": "NULLABLE", "description": "int_field"},
{"name": "int_field_repeated", "type": "INTEGER", "mode": "REPEATED", "description": "int_field_repeated"}
]
wrapper_json = {'actual_output': str(json.dumps(bigquery_schema))}
print(json.dumps(wrapper_json))
which I can then access within terraform
# main.tf
data "external" "bigquery_schema" {
program = ["python", "${path.module}/get_dynamic_bigquery_schema.py"]
}
locals {
bigquery_schema= data.external.bigquery_schema.result['actual_output']
}

jq construct with value strings spanning multiple lines

I am trying to form a JSON construct using jq that should ideally look like below:-
{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": [
"event"
],
"traffic_including": [
"unattributed_traffic"
],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
"attribution_campaign_id",
"attribution_creative",
"attribution_date_adjusted",
"attribution_date_utc",
"attribution_matched_by",
"attribution_matched_to",
"attribution_network",
"attribution_network_id",
"attribution_seconds_since",
"attribution_site_id",
"attribution_site_id",
"attribution_tier",
"attribution_timestamp",
"attribution_timestamp_adjusted",
"attribution_tracker",
"attribution_tracker_id",
"attribution_tracker_name",
"count",
"custom_dimensions",
"device_id_adid",
"device_id_android_id",
"device_id_custom",
"device_id_idfa",
"device_id_idfv",
"device_id_kochava",
"device_os",
"device_type",
"device_version",
"dimension_count",
"dimension_data",
"dimension_sum",
"event_name",
"event_time_registered",
"geo_city",
"geo_country",
"geo_lat",
"geo_lon",
"geo_region",
"identity_link",
"install_date_adjusted",
"install_date_utc",
"install_device_version",
"install_devices_adid",
"install_devices_android_id",
"install_devices_custom",
"install_devices_email_0",
"install_devices_email_1",
"install_devices_idfa",
"install_devices_ids",
"install_devices_ip",
"install_devices_waid",
"install_matched_by",
"install_matched_on",
"install_receipt_status",
"install_san_original",
"install_status",
"request_ip",
"request_ua",
"timestamp_adjusted",
"timestamp_utc"
]
}
What I have tried unsuccessfully thus far is below:-
json_construct=$(cat <<EOF
{
"api_key": "6AEC90B5-4169-59AF-7AC9-D655F83B4825",
"app_guid": "komacca-s-rewards-app-au-ios-production-cv8tx71",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv"
"columns_order": ["attribution_attribution_action","attribution_campaign","attribution_campaign_id","attribution_creative","attribution_date_adjusted","attribution_date_utc","attribution_matched_by","attribution_matched_to","attributio
network","attribution_network_id","attribution_seconds_since","attribution_site_id","attribution_tier","attribution_timestamp","attribution_timestamp_adjusted","attribution_tracker","attribution_tracker_id","attribution_tracker_name","
unt","custom_dimensions","device_id_adid","device_id_android_id","device_id_custom","device_id_idfa","device_id_idfv","device_id_kochava","device_os","device_type","device_version","dimension_count","dimension_data","dimension_sum","ev
t_name","event_time_registered","geo_city","geo_country","geo_lat","geo_lon","geo_region","identity_link","install_date_adjusted","install_date_utc","install_device_version","install_devices_adid","install_devices_android_id","install_
vices_custom","install_devices_email_0","install_devices_email_1","install_devices_idfa","install_devices_ids","install_devices_ip","install_devices_waid","install_matched_by","install_matched_on","install_receipt_status","install_san_
iginal","install_status","request_ip","request_ua","timestamp_adjusted","timestamp_utc"]
}
EOF)
followed by:-
echo "$json_construct" | jq '.'
I get the following error:-
parse error: Expected separator between values at line 10, column 15
I am guessing it is because of the string literal which spans to multiple lines that jq is unable to parse it.
Use jq itself:
my_formatted_json=$(jq -n '{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
...,
"timestamp_utc"
]
}')
Your input "JSON" is not valid JSON, as indicated by the error message.
The first error is that a comma is missing after the key/value pair: "delivery_format": "csv", but there are others -- notably, JSON strings cannot be split across lines. Once you fix the key/value pair problem and the JSON strings that are split incorrectly, jq . will work with your text. (Note that once your input is corrected, the longest JSON string is quite short -- 50 characters or so -- whereas jq has no problems processing strings of length 10^8 quite speedily ...)
Generally, jq is rather permissive when it comes to JSON-like input, but if you're ever in doubt, it would make sense to use a validator such as the online validator at jsonlint.com
By the way, the jq FAQ does suggest various ways for handling input that isn't strictly JSON -- see https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json
Along the lines of chepner's suggestion since jq can read raw text data you could just use a jq filter to generate a legal json object from your script variables. For example:
#!/bin/bash
# whatever logic you have to obtain bash variables goes here
key=XXXXXXXXXX-7AC9-D655F83B4825
guid=XXXXXXXXXXXXXX
# now use jq filter to read raw text and construct legal json object
json_construct=$(jq -MRn '[inputs]|map(split(" ")|{(.[0]):.[1]})|add' <<EOF
api_key $key
app_guid $guid
EOF)
echo $json_construct
Sample Run (assumes executable script is in script.sh)
$ ./script.sh
{ "api_key": "XXXXXXXXXX-7AC9-D655F83B4825", "app_guid": "XXXXXXXXXXXXXX" }
Try it online!

Pass bash variable (string) as jq argument to walk JSON object

This is seemingly a very basic question. I am new to JSON, so I am prepared for the incoming facepalm.
I have the following JSON file (test-app-1.json):
"application.name": "test-app-1",
"environments": {
"development": [
"server1"
],
"stage": [
"server2",
"server3"
],
"production": [
"server4",
"server5"
]
}
The intent is to use this as a configuration file and reference for input validation.
I am using bash 3.2 and will be using jq 1.4 (not the latest) to read the JSON.
The problem:
I need to return all values in the specified JSON array based on an argument.
Example: (how the documentation and other resources show that it should work)
APPLICATION_ENVIRONMENT="developement"
jq --arg appenv "$APPLICATION_ENVIRONMENT" '.environments."$env[]"' test-app-1.json
If executed, this returns null. This should return server1.
Obviously, if I specify text explicitly matching the JSON arrays under environment, it works just fine:
jq 'environments.development[]' test-app-1.json returns: server1.
Limitations: I am stuck to jq 1.4 for this project. I have tried the same actions in 1.5 on a different machine with the same null results.
What am I doing wrong here?
jq documentation: https://stedolan.github.io/jq/manual/
You have three issues - two typos and one jq filter usage issue:
set APPLICATION_ENVIRONMENT to development instead of developement
use variable name consistently: if you define appenv, use $appenv, not $env
address with .environments[$appenv]
When fixed, looks like this:
$ APPLICATION_ENVIRONMENT="development"
$ jq --arg appenv "$APPLICATION_ENVIRONMENT" '.environments[$appenv][]' test-app-1.json
"server1"

Need help! - Unable to load JSON using COPY command

Need your expertise here!
I am trying to load a JSON file (generated by JSON dumps) into redshift using copy command which is in the following format,
[
{
"cookieId": "cb2278",
"environment": "STAGE",
"errorMessages": [
"70460"
]
}
,
{
"cookieId": "cb2271",
"environment": "STG",
"errorMessages": [
"70460"
]
}
]
We ran into the error - "Invalid JSONPath format: Member is not an object."
when I tried to get rid of square braces - [] and remove the "," comma separator between JSON dicts then it loads perfectly fine.
{
"cookieId": "cb2278",
"environment": "STAGE",
"errorMessages": [
"70460"
]
}
{
"cookieId": "cb2271",
"environment": "STG",
"errorMessages": [
"70460"
]
}
But in reality most JSON files from API s have this formatting.
I could do string replace or reg ex to get rid of , and [] but I am wondering if there is a better way to load into redshift seamlessly with out modifying the file.
One way to convert a JSON array into a stream of the array's elements is to pipe the former into jq '.[]'. The output is sent to stdout.
If the JSON array is in a file named input.json, then the following command will produce a stream of the array's elements on stdout:
$ jq ".[]" input.json
If you want the output in jsonlines format, then use the -c switch (i.e. jq -c ......).
For more on jq, see https://stedolan.github.io/jq

Why won't jsontool parse my array?

I'm having trouble parsing this json, which contains an array as its outermost element:
response=[ { "__type": "File", "name": "...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg", "url": "http://files.parse.com/ac3f079b-cacb-49e9-bd74-8325f993f308/...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg" } ]
for blob in $response
do
url=$(echo $blob | json url)
done
But this last json parsing gives a bunch of errors:
json: error: input is not JSON: Bad object at line 2, column 1:
^
Do I need to do something special to turn a JSON array into a bash array, or vice versa?
You should quote the value of reponse to protect it from the shell trying to interpret it:
response='[ { "__type": "File", "name": "...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg", "url": "http://files.parse.com/ac3f079b-cacb-49e9-bd74-8325f993f308/...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg" } ]'
And you can't expect the shell to be able to parse JSON, so for blob in $response isn't going to work out. According to http://trentm.com/json/, using -a should handle the array:
while read url ; do
# use $url here...
done < <(echo "$response" | json -a url)
After putting your json in a file and piping it through jsontool it works just fine, so I'm guessing your file has some weird whitespace which is breaking in jsontool.
Try this instead:
cat your_file.json | sed 's/[[:space:]]\+$//' | json