How to configure LumberJack(LogStash-forwarder) on windows - json

I've installed ELK on my ubuntu server using this manual,
Now i want to index some log files from a windows server so I installed a logstash forwarder (LumberJack), but I can't get it to run.
this is the logstash-forwarder.conf file :
{
"network": {
"servers": [ "http://XX.XX.XX.XX:5000" ],
"ssl key": "D:/lumberjack/pki/tls/certs/logstash-forwarder.crt",
"ssl ca": "D:/lumberjack/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15,
},
"files": [
{
"paths": [
#single paths are fine
"D:/bea12/Oracle/Middleware/domains/Google/servers/RT1/logs/AppLogs/RT1_APP_9_0.log",
#globs are fine too, they will be periodically evaluated
#to see if any new files match the wildcard.
"/var/logauth.log"
],
]
}
}
and this is the Error I get when i'm trying to run the "lumberjack.exe" ,
That I created with go-build:
2015/04/30 18:17:39.052033 Failed unmarshalling json: invalid character '}' looking for beginning of object key string
2015/04/30 18:17:39.052033 Could not load config file d:\lumberjack\logstash-forwarder.conf: invalid character '}' looking for beginning of object key string
Can anyone please tell me what am I doing wrong?
By the way this is the command I'm using to run the forwarder:
lumberjack.exe -config="d:\lumberjack\logstash-forwarder.conf"

Ok.
So the problem was in the configuration file, There were 2 unnecesary commas and no need for the http:\ at the start:
{
"network": {
"servers": [ "XX.XX.XX.XX:5000" ],
"ssl key": "D:/lumberjack/pki/tls/certs/logstash-forwarder.key",
"ssl ca": "D:/lumberjack/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
#single paths are fine
"D:/bea12/Oracle/Middleware/domains/google/servers/RT1/logs/AppLogs/RT1_APP_9_0.log",
#globs are fine too, they will be periodically evaluated
#to see if any new files match the wildcard.
"/var/logauth.log"
]
}
]
}
This is my suggested configuration file for LumberJack on windows.

Related

Getting Error: Failed to parse Dockerrun JSON file: json: invalid use of ,string struct tag, trying to unmarshal unquoted value into int

I used JSON linter to check the validity of this json and it says ok. But when I deploy this to AWS Elastic Beanstalk, it errors out with message:
Error: Failed to parse Dockerrun JSON file: json: invalid use of
,string struct tag, trying to unmarshal unquoted value into int
Below is Dockerrun.aws.json:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "delivery-slot-notifier",
"image": "akshaylokur/delivery-slot-notifier:latest",
"essential": true,
"portMappings": [
{
"hostPort": 8080,
"containerPort": 8080
}
],
"memory": 128
}
]
}
Any clues?
Thanks
You should be seeing that error in your logs because your Dockerrun.aws.json has a bunch of options are not supported by Beanstalk (looks like you are using multi container docker-run for single container?). Also, for single docker env, Version has to be 1, whereas 2 is used for multi container env. Here's a Dockerrun.aws.json that will work:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "akshaylokur/delivery-slot-notifier:latest",
"Update": "true"
},
"Ports": [
{
"hostPort": 8080,
"containerPort": 8080
}
]
}
Here are all the supported options for Dockerrun.aws.json

ambari rest API + set json configuration in ambari

To create a new config group, it is mandatory to provide a config group name, a tag and the cluster name to which it belongs. The tag as seen in this example is a name of the service. Two config groups with the same tag cannot be associated with the same host.
how to run the following json file with curl ?
in order to set this config group in ambari
POST /api/v1/clusters/c1/config_groups
[
{
"ConfigGroup": {
"cluster_name": "c1",
"group_name": "hdfs-nextgenslaves",
"tag": "HDFS",
"description": "HDFS configs for rack added on May 19, 2010",
"hosts": [
{
"host_name": "host1"
}
],
"desired_configs": [
{
"type": "core-site",
"tag": "nextgen1",
"properties": {
"key": "value"
}
}
]
}
}
]
 reference - https://github.com/swagle/test/blob/master/docs/api/v1/config-groups.md
Is your question about how to send multiline json with curl? You can find different methods here.

How to use user variables with file provisioner in Packer?

I have a packer json like:
"builders": [{...}],
"provisioners": [
{
"type": "file",
"source": "packer/myfile.json",
"destination": "/tmp/myfile.json"
}
],
"variables": {
"myvariablename": "value"
}
and myfile.json is:
{
"var" : "{{ user `myvariablename`}}"
}
The variable into the file does get replaced, is a sed replacement with shell provisioner after the file the only option available here?
Using packer version 0.12.0
You have to pass these as environment variables. For example:
"provisioners": [
{
"type": "shell"
"environment_vars": [
"http_proxy={{user `proxy`}}",
],
"scripts": [
"some_script.sh"
],
}
],
"variables": {
"proxy": null
}
And in the script you can use $http_proxy
So far I've come just with the solution to use file & shell provisioner. Upload file and then replace variables in file via shell provisioner which can be fed from template variables provided by e.g. HashiCorp Vault
Yo may use OS export function to set environment and pass it to Packer
Here is a config using OS ENV_NAME value to choose local folder to copy from
export ENV_NAME=dev will set local folder to dev
{
"variables": {
...
"env_folder": "{{env `ENV_NAME`}}",
},
"builders": [{...}]
"provisioners": [
{
"type": "file",
"source": "files/{{user `env_folder`}}/",
"destination": "/tmp/"
},
{...}
]
}
User variables must first be defined in a variables section within your template. Even if you want a user variable to default to an empty string, it must be defined. This explicitness helps reduce the time it takes for newcomers to understand what can be modified using variables in your template.
The variables section is a key/value mapping of the user variable name to a default value. A default value can be the empty string. An example is shown below:
{
"variables": {
"aws_access_key": "",
"aws_secret_key": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
// ...
}]
}
check this link for more information

swagger-codegen: errors with \d+ patterns and type: "array"

I'm a newbie to Swagger. I've used the swagger servlet to generate my swagger.json file from our REST API Java classes. The swagger.json file shows swagger 2.0 (I assume this is the 2.0 schema version). There was nothing fancy in the source files, just #Api and a few #ApiOperation annotations.
Then I tried using swagger-codegen-cli (both version 2.1.4 and 2.1.6-SNAPSHOT, the latest) to generate HTML output from the JSON file. I got the following results on both:
reading from dsm_swagger.json
[main] ERROR io.swagger.codegen.DefaultCodegen - unexpected missing property for name suppressed
[main] WARN io.swagger.codegen.DefaultCodegen - skipping invalid property {
"type" : "array"
}
writing file /home/combs/dsm_swagger/./index.html
So I get an output file, but any types that are flagged as lists of objects are not handled correctly. These do appear to be valid 2.0 constructs.
I'm also getting Jackson errors about invalid escape characters because it sees
"pattern": "\d+"
in the file. I can work around the \d by using [0-9], but assume it should be handled as is.
Has anybody seen these particular issues and know if they're either fixed or there is a workaround in swagger-codegen or the source file? Is swagger-codegen actually handling v2.0 specs correctly? Any pointers to up to date info or code would be appreciated!
EDIT:
As mentioned in a comment, by using "#JsonIgnore" and "#JsonProperty" in appropriate places and upgrading to V1.5.6 of swagger-core, I got around the issues with invalid property and type "array" messages. Here's an example of the issue with \d:
"/v1/admins/{adminId}": {
"put": {
"tags": [
"admins"
],
"summary": "Update information about a particular admin, given its ID. The update information is passed in the POST body.",
"description": "Longer notes about what this does",
"operationId": "updateUser",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"parameters": [
{
"name": "adminId",
"in": "path",
"required": true,
"type": "integer",
"pattern": "\d+",
"format": "int64"
},
{
"in": "body",
"name": "body",
"required": false,
"schema": {
"$ref": "#/definitions/UserUpdateInfo"
}
}
],
"responses": {
"200": {
"description": "successful operation",
"schema": {
"$ref": "#/definitions/UserInfo"
}
}
}
}
},
This is the exact output of swagger-core, and yet swagger-codegen fails with the following:
combs#dcombs-lap:~/dsm_swagger$ gen_file
reading from dsm_swagger.json
reading from dsm_swagger.json
com.fasterxml.jackson.core.JsonParseException: Unrecognized character escape 'd' (code 100)
at [Source: dsm_swagger.json; line: 411, column: 27]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508)
at com.fasterxml.jackson.core.base.ParserMinimalBase._handleUnrecognizedCharacterEscape(ParserMinimalBase.java:485)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeEscaped(UTF8StreamJsonParser.java:2924)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2209)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2165)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:279)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:224)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:262)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:221)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:218)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:218)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:218)
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:62)
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:14)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3066)
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:1861)
at io.swagger.parser.SwaggerCompatConverter.readResourceListing(SwaggerCompatConverter.java:139)
at io.swagger.parser.SwaggerCompatConverter.read(SwaggerCompatConverter.java:74)
at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:73)
at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:317)
at io.swagger.codegen.cmd.Generate.run(Generate.java:186)
at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)
Exception in thread "main" java.lang.RuntimeException: missing swagger input or config!
at io.swagger.codegen.DefaultGenerator.generate(DefaultGenerator.java:89)
at io.swagger.codegen.cmd.Generate.run(Generate.java:188)
at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)
combs#dcombs-lap:~/dsm_swagger$

OrientDB ETL Throws exception on loading CSV file

I am trying to load a simple CSV file to OrientDB and it always throws this exception.
CSV File Content
id, name, role
1, Sarath, Architect
2, Anoop, Project Manager
3, Nazeem, Lead Developer
4, Rini, Senior Developer
5, Shine, iOS Developer
6, Vishnu, iOS Developer
json config file
{
"source": { "file": { "path": "./dev.csv" } },
"extractor": { "row": {} },
"transformers": [
{
"csv": {}
},
{ "vertex": { "class": "Person" } }
],
"loader": {
"orientdb": {
"dbURL": "remote:localhost/dev",
"dbType": "graph",
"dbUser": "root",
"dbPassword": "root",
"dbAutoCreate": true,
"classes": [
{"name": "Person", "extends": "V"}
], "indexes": [
{"class":"Person", "fields":["id:integer"], "type":"UNIQUE" }
]
}
}
}
All the time it shows this exception. I tried different CSV options but it did not work. Looks like the file reading itself throwing exception.
» oetl posts.json sarat#Saraths-MacBook-Air
OrientDB etl v.2.0.1 (build #BUILD#) www.orientechnologies.com
Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on loading config file: posts.json
at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:151)
Platform Details
OS X 10.10.2
java version "1.8.0_25"
OrientDB - v.2.0.1
The stack trace gives you the cause of the problem (you only showed the first, not interesting lines in your example).
Check:
The server is running (you specified remote:localhost/dev, so the server must be running on the default port at the same machine).
The database exists and has "root" / "root" as credentials (note that mostly "admin" "admin" is used)
The file dev.csv is present in the current directory
Tried out your example and works without modification.
I had the same problem, and it was due to one of the directories in the path to the JSON config file having a space in it. The oetl script doesn't handle this well and thus can't find the script.
#rmuller: The OP showed the entire traceback. That's all you get in this case.