I'm trying to use only one file in my Data Bag to contain multiple items (specifically for the users cookbook). All samples show isolated JSON objects. How do I store multiple ones in one file?
I've tried
{
"id": "1"
},
{
"id": "2"
}
and
[
{
"id": "1"
},
{
"id": "2"
}
]
and
{
"id": "1"
}
{
"id": "2"
}
I'm getting:
Net::HTTPFatalError: users_manage[xxxxx] (common::default line 76) had an error: Net::HTTPFatalError: 500 "Internal Server Error"
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.8.1/lib/chef/http.rb:145:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.8.1/lib/chef/http.rb:110:in `get'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.8.1/lib/chef/search/query.rb:158:in `call_rest_service'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.8.1/lib/chef/search/query.rb:87:in `search'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.8.1/lib/chef/dsl/data_query.rb:39:in `search'
I'm using chef-solo -z with the chef-solo-search cookbook.
Can't I pass an array of items from one file?
You do not store multiple items in one data bag file in Chef. Just specify it in separate files. This is, how Chef works.
Can you try something like:
{
"id": "users",
"users": [
{ ... user 1 info ... },
{ ... user 2 info ... },
{ ... user 3 info ... },
...
]
}
Related
I am trying to do some work with log visualization tools (Elastic and/or Splunk), but first I need to produce and format the log files from a simulation I am writing. My question, which I can't seem to find clear guidance on is:
How to store multiple, what I believe are root element JSON entries in a single text file
How to work with nested JSON structures
I am ultimately trying to have every entry follow the same form:
{"entry_id": 1,
"TIME": "12:00:12Z012/01/2022",
"LOG_TYPE":"ERROR_REPORT",
"DATA": {
"FIELD A" : "ABC",
"FIELD B" : "DEF"
}
},
{"entry_id": 2,
"TIME": "12:15:12Z012/01/2022",
"LOG_TYPE":"STATUS_REPORT",
"DATA": {
"FIELD C" : "HIJ",
"FIELD D" : 123
}
}
Some options I saw
Use an array []
Use NDJSON
Use some log template??
Any insight would be helpful
JSON files need to be a single object and can't be INVALID themselves.
Option 1: Create a single file for each of the objects, using a numeric naming system for the files, then iterating over the files in your method.
Option 2: Create a single file but have each entry contained in an array eg:
{
"entries": [
{
"entry_id": 1,
"TIME": "12:00:12Z012/01/2022",
"LOG_TYPE": "ERROR_REPORT",
"DATA": {
"FIELD A": "ABC",
"FIELD B": "DEF"
}
},
{
"entry_id": 2,
"TIME": "12:15:12Z012/01/2022",
"LOG_TYPE": "STATUS_REPORT",
"DATA": {
"FIELD C": "HIJ",
"FIELD D": 123
}
}
]
}
When I run the jq command to parse a json document from the amazon cli I have the following problem.
I’m parsing through the IP address and a tag called "Enviroment". The enviroment tag in the instance does not exist therefore it does not throw me any result.
Here's an example of the relevant output returned by the AWS CLI
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
I’m running the following command
aws ec2 describe-instances --filters "Name=tag:Name,Values=Balance-OTA-SS_a" | jq -c '.Reservations[].Instances[] | ({IP: .PrivateIpAddress, Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)})'
## output
empty
How do I show the IP address in the output of the command even if the enviroment tag does not exist?
Regards,
Let's assume this input:
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
This is the format returned by describe-instances, but with all the irrelevant fields removed.
Note that tags is always a list of objects, each of which has a Key and a Value. This format is perfect for from_entries, which can transform this list of tags into a convenient mapping object. Try this:
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags|from_entries.Environment)
}
{"IP":"10.0.0.1","Ambiente":"alpha"}
{"IP":"10.0.0.2","Ambiente":null}
That answers how to do it. But you probably want to understand why your approach didn't work.
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)
}
The .[] filter you're using on the tags can return zero or multiple results. Similarly, the select filter can eliminate some or all items. When you apply this inside an object constructor (the expression from { to }), you're causing that whole object to be created a variable number of times. You need to be very careful where you use these filters, because often that's not what you want at all. Often you instead want to do one of the following:
Wrap the expression that returns multiple results in an array constructor [ ... ]. That way instead of outputting the parent object potentially zero or multiple times, you output it once containing an array that potentially has zero or multiple items. E.g.
[.Tags[]|select(.Key=="Environment")]
Apply map to the array to keep it an array but process its contents, e.g.
.Tags|map(select(.Key=="Environment"))
Apply first(expr) to capture only the first value emitted by the expression. If the expression might emit zero items, you can use the comma operator to provide a default, e.g.
first((.Tags[]|select(.Key=="Environment")),null)
Apply some other array-level function, such as from_entries.
.Tags|from_entries.Environment
You can either use an if ... then ... else ... end construct, or //. For example:
.Reservations[].Instances[]
| {IP: .PrivateIpAddress} +
({Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)}
// null)
I'm relatively new to Power Query, but I'm pulling in this basic structure of JSON from a web api
{
"report": "Cost History",
"dimensions": [
{
"time": [
{
"name": "2019-11",
"label": "2019-11",
…
},
{
"name": "2019-12",
"label": "2019-12",
…
},
{
"name": "2020-01",
"label": "2020-01",
…
},
…
]
},
{
"Category": [
{
"name": "category1",
"label": "Category 1",
…
},
{
"name": "category2",
"label": "Category 2",
…
},
…
]
}
],
"data": [
[
[
40419.6393798211
],
[
191.44
],
…
],
[
[
2299.652439184997
],
[
0.0
],
…
]
]
}
I actually have 112 categories and 13 "times". I figured out how to do multiple queries to turn the times into column headers and the categories into row labels (I think). But the data section is alluding me. Because each item is a list within a list I'm not sure how to expand it all out. Each object in the date array will have 112 numbers and there will be 13 objects. If that all makes sense.
So ultimately I want to make it look like
2019-11 2019-20 2020-01 ...
Category 1 40419 2299
Category 2 191 0
...
First time asking a question on here, so hopefully this all makes sense and is clear. Thanks in advance for any help!
i am also researching this exact thing and looking for a solution. In PQ, it displays nested arrays as a list and there is a function to extract values choosing a separating characterenter image description here
So this becomes, this
enter image description here
= Table.TransformColumns(#"Filtered Rows", {"aligned_to_ids", each Text.Combine(List.Transform(_, Text.From), ","), type text})
However the problem i'm trying to solve is when the nested json has multiple values like this: enter image description here
And when these LIST are extracted then an error message is caused, = Table.TransformColumns(#"Extracted Values1", {"collaborators", each Text.Combine(List.Transform(_, Text.From), ","), type text})
Expression.Error: We cannot convert a value of type Record to type Text.
Details:
Value=
id=15890
goal_id=323
role_id=15
Type=[Type]
It seems the multiple values are not handled and PQ does not recognise the underlying structure to enable the columns to be expanded.
We have a heavily nested json document containing server metrcs, the document contains > 1000 fields some of which are completely irrelevant to us for analytic purposes so i would like to remove them before indexing the document in Elastic.
However i am unable to find the correct filter to use as the fields i want to remove have common names in multiple different objects within the document.
The source document looks like this ( reduced in size for brevity)
[
{
"server": {
"is_master": true,
"name": "MYServer",
"id": 2111
},
"metrics": {
"Server": {
"time": {
"boundary": {},
"type": "TEXT",
"display_name": "Time",
"value": "2018-11-01 14:57:52"
}
},
"Mem_OldGen": {
"used": {
"boundary": {},
"display_name": "Used(mb)",
"value": 687
},
"committed": {
"boundary": {},
"display_name": "Committed(mb)",
"value": 7116
}
"cpu_count": {
"boundary": {},
"display_name": "Cores",
"value": 4
}
}
}
}
]
The data is loaded into logstash using the http_poller input plugin and needs to be processed before sending to Elastic for indexing.
I am trying to remove the fields that are not relevant for us to track for analytical purposes, these include the "display_name" and "boundary" fields from each json object in the different metrics.
I have tried using the mutate filter to remove the fields but because they exist in so many different objects it requires to many coded paths to be added to the logstash config.
I have also looked at the ruby filter, which seems promising as it can look the event, but i am unable to get it to crawl the entire json document, or more importantly actually remove the fields.
Here is what i was trying as a test
filter {
split{
field => "message"
}
ruby {
code => '
event.get("[metrics][Mem_OldGen][used]").to_hash.keys.each { |k|
logger.info("field is:", k)
if k.include?("display_name")
event.remove(k)
end
if k.include?("boundary")
event.remove(k)
end
}
'
}
}
It first splits the input at the message level to create one event per server, then tries to remove the fields from a specific metric.
Any help you be greatly appreciated.
If I get the point, you want to keep just the value key.
So, considering the response hash:
response = {
"server": {
"is_master": true,
"name": "MYServer",
"id": 2111
},
"metrics": {
...
You could do:
response[:metrics].transform_values { |hh| hh.transform_values { |h| h.delete_if { |k,v| k != :value } } }
#=> {:server=>{:is_master=>true, :name=>"MYServer", :id=>2111}, :metrics=>{:Server=>{:time=>{:value=>"2018-11-01 14:57:52"}}, :Mem_OldGen=>{:used=>{:value=>687}, :committed=>{:value=>7116}, :cpu_count=>{:value=>4}}}}
Lets take a look at google example for simple array
"chats": {
"one": {
"title": "Historical Tech Pioneers",
"lastMessage": "ghopper: Relay malfunction found. Cause: moth.",
"timestamp": 1459361875666
},
"two": { ... },
"three": { ... }
},
My question is why do we have to put name for each chat ? if i want to add another object i need to know which are the existing keys and create new one for which reason ? i just want a list of chats in this example and the names one two three means nothing to me...
why cant i put a list of objects with no name under chats ?
"chats": {
"one": {
"title": "Historical Tech Pioneers",
"lastMessage": "ghopper: Relay malfunction found. Cause: moth.",
"timestamp": 1459361875666
},
"two": { ... },
"three": { ... }
},
Instead of one, two and three, you can use random ids using the push() method in android, so you will have something like this:-
"chats": {
"id_1": {
"title": "Historical Tech Pioneers",
"lastMessage": "ghopper: Relay malfunction found. Cause: moth.",
"timestamp": 1459361875666
},
"id_2": { ... },
"id_3": { ... }
},
You can retrieve the key like this:-
String mGroupId = mGroupRef.push().getKey();
You can think of the ids as primary key