I am using fluentd out_webhdfs to post data into hadoop hdfs
td-agent.conf:
<match access.**>
type webhdfs
host hadoop1
port 50070
path /data/access.log.%Y%m%d_%H.log
output_data_type ltsv
field_separator TAB
output_include_tag false
</match>
using curl command to post data:
curl -X POST -d 'json={"name":"abc","id":"8754738"}' http://localhost:8888/.access
the result in hdfs file is below:
2014-08-20T08:25:28Z name:abc id:8754738
Is there any way to remove the "name:" and "id:" out , the result I want is :
2014-08-20T08:25:28Z abc 8754738
Any body help me please !
Related
I tried to extract data from below cbq command which was successful.
cbq -u Administrator -p Administrator -e "http://localhost:8093" --script= SELECT * FROM `sample` where customer.id=="12345'" -q | jq '.results' > temp.json;
However when I am trying to import the same data in json format to target cluster using below command I am getting error.
cbimport json -c http://{target-cluster}:8091 -u Administrator -p Administrator -b sample -d file://C:\Users\{myusername}\Desktop\temp.json -f list -g %docId%
JSON import failed: 0 documents were imported, 0 documents failed to be imported
JSON import failed: input json is invalid: ReadArray: expect [ or , or ] or n, but found {, error found in #1 byte of ...|{
"requ|..., bigger context ...|{
"requestID": "2fc34542-4387-4643-8ae3-914e316|...],```
```{
"requestID": "6ef38b8a-8e70-4c3d-b3b4-b73518a09c62",
"signature": {
"*": "*"
},
"results": [
{
"{Bucket-name}":{my-data}
"status": "success",
"metrics": {
"elapsedTime": "4.517031ms",
"executionTime": "4.365976ms",
"resultCount": 1,
"resultSize": 24926
}
It looks like the file which was extracted from cbq command has control fields details like RequestID, metrics, status etc. Also json in pretty format. If I manually remove it(remove all fields except {my-data}) then put in a json file and make json unpretty then it works. But I want to automate it in a single run. Is there a way to do it in cbq command.
I don't find any other utility or way to use where condition on cbexport to do that on Couchbase, because the document which are exported using cbexport can be imported using cbimport easily.
For the cbq command, you can use the --quiet option to disable the startup connection messages and the --pretty=false to disable pretty-print. Then, to extract just the documents in cbimport json lines format, I used jq.
This worked for me -- selecting documents from travel-sample._default._default (for the jq filter, where I have _default, you would put the Bucket-name, based on your example):
cbq --quiet --pretty=false -u Administrator -p password --script='select * from `travel-sample`._default._default' | jq --compact-output '.results|.[]|._default' > docs.json
Then, importing into test-bucket1:
cbimport json -c localhost -u Administrator -p password -b test-bucket1 -d file://./docs.json -f lines -g %type%_%id%
cbq documentation: https://docs.couchbase.com/server/current/tools/cbq-shell.html
cbimport documentation: https://docs.couchbase.com/server/current/tools/cbimport-json.html
jq documentation:
https://stedolan.github.io/jq/manual/#Basicfilters
I am new to AzureCLI script. I request some guidance here:
Please find the secret(masked value) from my Azure key vault.
{
"**attributes**": {
"created": "2021-10-23T04:26:19+00:00",
"enabled": true,
"id": "https://app-kv-axderfctt.vault.azure.net/secrets/connectionstr/cbcncbc",
"name": "connectionstr",
"**value**": "\"{\\\"eventHubNameSpace\\\":\\\"Pji-11111111-132d-46f7-af0f-aq1234\\\",\\\"eventHubName\\\":\\\"pipeline-swswsw-fb95-4e16-8364-oiu8ikji\\\",\\\"**ConnectionString**\\\":\\\"Endpoint=sb://pji-uit8tihj-132d-46f7-af0f-89897899.servicebus.windows.net/;SharedAccessKeyName=343456-ddddd-4720-b3db-oi89087;SharedAccessKey=xxxx+xxxxxxxx=;EntityPath=pipeline-ttrgfrtyh-fb95-4e16-8364-897uy678i\\\"}\""
I am able to get the complete value but not the specific property(ConnectionString) using below command
CONNECTION_STRING=$(az keyvault secret show --vault-name ${{ parameters.KeyVault }} --name ${{ parameters.SecretName}}--query 'value.ConnectionString')
echo "$CONNECTION_STRING"
I get below error:
ERROR: incorrect usage: [Required] --value VALUE | --file PATH
Please advise on how to extract ConnectionString value excluding EntityPath by escaping slashes \\\.
Thanks in advance.
I resolved this issue using jq library
task: Bash#3
displayName: "Parse WarmRepo Connection String"
inputs:
targetType: 'inline'
script: |
# Parse WarmRepo Connection String
echo 'Parse WarmRepo Connection String'
#Refer https://stedolan.github.io/jq/ to know about jq queries
PipelineConnectionString=`jq -r '.ConnectionString' <<<'$(${{ parameters.StreamingPipelineName }}-EventHub-connectionString)'`
I am trying to get data with rundeck webhook plugin, and for this i am usig curl command:
curl -X POST -d '{"name":"John", "age":30, "car":null}' https://rundeck_server/api/12/webhook/QSxTDYd08dcYxKh1R5YJNOPQvmSJH2Z8#Netbox_Job
In rundeck webhook plugin options i add those 2 variables, 'whkpayload' to get all the json data and 'name' to get the name only (must return John in this example):
-whkpayload ${raw} -name ${data.name}
And finally i show them with those lines:
echo #option.whkpayload#
echo #option.name#
I get an empty result and i can't figure out why. Any one may help me please ?
Following this, you need to use an option called whkpayload in your job, and set it as ${raw} in the webhook configuration.
I made an example:
The job definition in YAML format (with the whkpayload option):
- defaultTab: nodes
description: ''
executionEnabled: true
id: 0fcfca07-02f6-4583-a3eb-0002276bdf2d
loglevel: INFO
name: HelloWorld
nodeFilterEditable: false
options:
- name: age
- name: car
- name: name
- name: whkpayload
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- description: command step
exec: echo "name ${option.name} - age ${option.age} - car ${option.car} - payload
${option.whkpayload}"
- description: inline-script step
fileExtension: .sh
interpreterArgsQuoted: false
script: echo "name #option.name# - age #option.age# - car #option.car# - payload
#option.whkpayload#"
scriptInterpreter: /bin/bash
keepgoing: false
strategy: node-first
uuid: 0fcfca07-02f6-4583-a3eb-0002276bdf2d
The webhook configuration.
The webhook calling from cURL:
curl -H "Content-Type: application/json" -X POST -d '{"field1":"John", "field2":30, "field3":"chevy"}' http://localhost:4440/api/40/webhook/0vBZjWWrnXWvqENEdxkn0JRvjn5R63J0#MyWebhook
The result.
I have the following very simple manifest:
type: install
name: very simple manifest
onInstall:
- log: installing manifest
I can install it from the Jelastic Dashboard. There is an import function in the main menu where I can copy / paste that manifest content and it gets installed. In the Jelastic console, I can see
[15:36:38 manifest.settings]: BEGIN INSTALLATION: very simple manifest
[15:36:39 manifest.settings]: BEGIN HANDLE EVENT: {"topic":"application/install","envAppid":""}
[15:36:39 manifest.settings:1]:> installing manifest
[15:36:39 manifest.settings]: END HANDLE EVENT: application/install
[15:36:39 manifest.settings]: END INSTALLATION: very simple manifest
and the Jelastic dashboard confirms installation.
Now, when I do the same, but via the Jelastic REST API, i.e. using the endpoint
http://my-jelastic-provide.com/1.0/marketplace/jps/REST/install
with the relevant data, then, it doesn't install. Instead, I get the strange error message
Can\'t find environment by domain [jelasticclient-master-0954606]
where jelasticclient-master-0954606 is the envName I set.
However, if I change my manifest to e.g.
type: install
name: very simple manifest
nodes:
count: 1
cloudlets: 4
nodeGroup: cp
image: alpine:latest
skipNodeEmails: true
onInstall:
- log: installing manifest
then it installs perfectly. What am I missing?
I am using Jelastic v6.0.2.
Your "very simple manifest" doesn't suppose any environment name to be passed.
That's why when you pass it you get an error "Can't find environment by domain [domain-name]" (Example1).
When you don't have the "nodes" parameter in the manifest (as in your second example), you shouldn't pass any environment name (Example2) or should pass the existing environment name (response is in Example3).
Example1:
curl -X POST 'https://jca.host-domain/1.0/marketplace/jps/rest/install' \
-d 'envName=jelasticclient-master-0954606' \
-d session=*** \
-d skipNodeEmails=1 \
-d ownerUid=UID \
--data-urlencode 'jps={ "type": "install", "name": "very simple manifest", "onInstall": [ { "log": "installing manifest" } ] }'
The response is:
{"result":11,"response":{"result":11,"source":"JEL","error":"domain [jelasticclient-master-0954606] doesn't exist"},"source":"JEL","error":"domain [jelasticclient-master-0954606] doesn't exist"}
When the environment name is not passed (Example2),
curl -X POST 'https://jca.host-domain/1.0/marketplace/jps/rest/install' \
-d session=*** \
-d skipNodeEmails=1 \
-d ownerUid=UID \
--data-urlencode 'jps={ "type": "install", "name": "very simple manifest", "onInstall": [ { "log": "installing manifest" } ] }'
the response is
{"result":0,"uniqueName":"3c819586-2ef7-4691-9faa-d3059459d20e","response":{"result":0,"uniqueName":"3c819586-2ef7-4691-9faa-d3059459d20e","successText":"","appid":""},"appid":"","successText":""}
When the environment with envName=jelasticclient-master-0954606 already exists, the response of the same request from the Example1 is as this (Example3)
{"result":0,"uniqueName":"b52a8db9-8850-4b66-958a-3dee3345b923","response":{"result":0,"uniqueName":"b52a8db9-8850-4b66-958a-3dee3345b923","successText":"","appid":"7b0c465f6c9573b8d8ce3ed59591781b"},"appid":"7b0c465f6c9573b8d8ce3ed59591781b","successText":""}
In other words, if you pass the environment name when deploying this "very simple manifest" this manifest is installed like an add-on because there is no "nodes" parameter in it but there is no existing environment "jelasticclient-master-0954606" to install this "add-on".
I am using Ansible to test some APIs. Instead of using get_url or uri module of ansible, i am using Curl requests with command/shell module.
I would need to extract a particular value from the json response of a curl api call, so that i can use that value in next curl request.
For example,
- name: Run a curl request to login and get jwt token
command: curl -k -X POST -H "Content-Type:application/json" --data '{"user":"{{username}}","password":"{{password}}"}' https://localhost:8001/api/v1/login
register: login_response
- name: Print the jwt response
debug:
var: login_response.stdout
- name: Run a curl request to get service token
command: curl -k -X GET -H "Accept:application/json" -H "Authorization:Bearer {{login_response.stdout.jwt}}" https://localhost:8001/api/v1/servicetoken/issue
register: service_token
- name: Print service token
debug:
var: service_token.stdout
The above ansible playbook will fail at third task because it cannot find a variable like:
login_response.stdout.jwt
Normally the variable in second task, ie, login_response.stdout will print something like this:
ok: [localhost] => {
"login_response.stdout": {
"jwt": "eyJhbGciOiJIUzUxMiIsImtpZCI6ImFiNmU0ZDg2LWE4YzgtNDU4OS04MmRiLWIxZTg1YzQwNDNlZiJ9.eyJleHAiOjE2MDU2MTIxODAuNjA5MTI1NCwiaWF0IjoxNjA1NjExODgwLjYwOTEyMC42MDkxMjU0fQ.ZC4a3H3j03ZmzDkjGj11cvxSls2qXZmVOGuIvKp8LHVpYOUyEJlWJJOTArHxKhxne3DsuqWoGpslR6KxuUOBFg",
"roles": [
{
"name": "Admin"
}
]
}
}
What i want is to extract that jwt token from the above response and use it in third task.
How can i achieve this?
I found the answer!
This can be achieved using from_json filter in order to make ansible able to access the json fields.
Now the working playbook should be like:
- name: Run a curl request to login and get jwt token
command: curl -k -X POST -H "Content-Type:application/json" --data '{"user":"{{username}}","password":"{{password}}"}' https://localhost:8001/api/v1/login
register: login_response
- name: Print the jwt token
debug:
var: (login_response.stdout|from_json).jwt
- name: Run a curl request to get service token
command: curl -k -X GET -H "Accept:application/json" -H "Authorization:Bearer {{(login_response.stdout|from_json).jwt}}" https://localhost:8001/api/v1/servicetoken/issue
register: service_token
- name: Print service token
debug:
var: (service_token.stdout|from_json).token
I refered here to get the idea: https://medium.com/#justinhennessy/parsing-json-with-ansible-bcbb8d4b6a54