I have updated the logging.properties file of tomcat to print the logs in json format.
But, the issue is value of "message " has some escape characters, which makes my log invalid json.
Please let me know how to escape these characters(:,[,],/) in json using default tomcat-juli.jar and used as a string.
Below is my updated logging.properties file:
handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, 3manager.org.apache.juli.AsyncFileHandler, 4host-manager.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler
.handlers = 1catalina.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################
java.util.logging.SimpleFormatter.format = {"date" :"%1$tF", "timestamp": "%1$tT" , "loggerlevel": "%4$s", "loggersource": "%3$s" , "message": "%5$s%6$s"}%n
1catalina.org.apache.juli.AsyncFileHandler.level = FINE
1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
1catalina.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
2localhost.org.apache.juli.AsyncFileHandler.level = FINE
2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
2localhost.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
3manager.org.apache.juli.AsyncFileHandler.level = FINE
3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
3manager.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
4host-manager.org.apache.juli.AsyncFileHandler.level = FINE
4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
4host-manager.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.AsyncFileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = 3manager.org.apache.juli.AsyncFileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers = 4host-manager.org.apache.juli.AsyncFileHandler
I would suggest you official way (without using middleware):
Download Elastic JUL formatter and logging core libraries:
https://mvnrepository.com/artifact/co.elastic.logging/jul-ecs-formatter
https://mvnrepository.com/artifact/co.elastic.logging/ecs-logging-core
Put them into tomcat/bin folder
List them in CLASSPATH system property in tomcat/bin/setenv.sh(bat).
If no such file - create it with the next content: on Unix -
export CLASSPATH="$CLASSPATH:ecs-logging-core.jar:jul-ecs-formatter.jar"
on Windows - set CLASSPATH=%CLASSPATH%;ecs-logging-core.jar;jul-ecs-formatter.jar
Edit tomcat/conf/logging.properties in such way:
handlers = java.util.logging.ConsoleHandler
.handlers = java.util.logging.ConsoleHandler
...
java.util.logging.ConsoleHandler.formatter = co.elastic.logging.jul.EcsFormatter
Such configuration will make tomcat to write all events in appropriate Elastic (Logstash) JSON format into console (stdout) which is very suitable when you running in docker or other container.
Documentation https://www.elastic.co/guide/en/ecs-logging/java/1.x/setup.html
I would do the following:
Download and build devatherock's jul json formatter
Download json-simple
Add both to your tomcat startup classpath in the $CATALINA_BASE/bin/setenv.sh script
In $CATALINA_BASE/conf/logging.properties, add or change the formatters for your log handlers to io.github.devatherock.json.formatter.JSONFormatter
Related
I created a packer json template on my local system with packer 1.7.7 installed.
Then I upgraded to hcl2 template. However, when I run the packer pipeline over the jenkins node having packer version 1.6.1. It throws this error:
Blocks of type "data" are not expected here.
Error: Unsupported block type
After researching, I realized that packer version 1.6.1 doesn't support data blocks in its templates, but it supports hcl2 templates.
Can anyone explain how I can replace the data block (ref template below) with something supported in packer version 1?
data "amazon-ami" "autogenerated_1"{
access_key = "${var.aws_access_key}"
filters = {
root-device-type = "ebs"
virtualization-type = "hvm"
name = "**** Linux *"
}
most_recent = true
region = "${var.aws_region}"
owners = ["${var.owner_id}"]
secret_key = "${var.aws_secret_key}"
}
when I am trying to consume this ami id in the source block It gives me error.
ami_name = "${var.ami_name}"
associate_public_ip_address = false
force_deregister = true
iam_instance_profile = "abc"
instance_type = "****"
region = "${var.aws_region}"
source_ami = data.amazon-ami.autogenerated_1.id
ssh_interface = "private_ip"
ssh_username = "user"
subnet_id = "subnet-********"
vpc_id = "vpc-***********"
}
The packer pipeline over the jenkins node having packer version 1.6.1.
Its not supported in such an old version. From docs:
Note: Data Sources is a feature included in Packer 1.7 and later
For my application, new file uploaded to storage is read and the data is added to a main file. The new file contains 2 lines, one a header and other an array whose values are separated by a comma. The main file will need maximum of 265MB. The new files will have maximum of 30MB.
def write_append_to_ecg_file(filename,ecg,patientdata):
file1 = open('/tmp/'+ filename,"w+")
file1.write(":".join(patientdata))
file1.write('\n')
file1.write(",".join(ecg.astype(str)))
file1.close()
def storage_trigger_function(data,context):
#Download the segment file
download_files_storage(bucket_name,new_file_name,storage_folder_name = blob_path)
#Read the segment file
data_from_new_file,meta = read_new_file(new_file_name, scale=1, fs=125, include_meta=True)
print("Length of ECG data from segment {} file {}".format(segment_no,len(data_from_new_file)))
os.remove(new_file_name)
#Check if the main ecg_file_exists
file_exists = blob_exists(bucket_name, blob_with_the_main_file)
print("File status {}".format(file_exists))
data_from_main_file = []
if ecg_file_exists:
download_files_storage(bucket_name,main_file_name,storage_folder_name = blob_with_the_main_file)
data_from_main_file,meta = read_new_file(main_file_name, scale=1, fs=125, include_meta=True)
print("ECG data from main file {}".format(len(data_from_main_file)))
os.remove(main_file_name)
data_from_main_file = np.append(data_from_main_file,data_from_new_file)
print("data after appending {}".format(len(data_from_main_file)))
write_append_to_ecg_file(main_file,data_from_main_file,meta)
token = upload_files_storage(bucket_name,main_file,storage_folder_name = main_file_blob,upload_file = True)
else:
write_append_to_ecg_file(main_file,data_from_new_file,meta)
token = upload_files_storage(bucket_name,main_file,storage_folder_name = main_file_blob,upload_file = True)
The GCF is deployed
gcloud functions deploy storage_trigger_function --runtime python37 --trigger-resource patch-us.appspot.com --trigger-event google.storage.object.finalize --timeout 540s --memory 8192MB
For the first file, I was able to read the file and write the data to the main file. But after uploading the 2nd file, its giving Function execution took 70448 ms, finished with status: 'connection error' On uploading the 3rd file, it gives the Function invocation was interrupted. Error: memory limit exceeded. Despite of deploying the function with 8192MB memory, I am getting this error. Can I get some help on this.
I'm a newcomer to VIM and installed GVIM successfully on my Windows 8.1 laptop. I uploaded the Syntastics plugin using Pathogen and, as I'm planning to write an ionic project, I also copied the tidy5 for html5 binary (from http://www.htacg.org/tidy-html5/) and referred to the binary in my VIMRC as follows:
" SYNTASTIC
set statusline+=%#warningmsg#
set statusline+=%{SyntasticStatuslineFlag()}
set statusline+=%*
" automatically load errors in the location list:
let g:syntastic_always_populate_loc_list = 1
let g:syntastic_auto_loc_list = 1
" check on errors when a file is loaded:
let g:syntastic_check_on_open = 1
" check on errors when saving the file:
let g:syntastic_check_on_wq = 0
let g:syntastic_html_tidy_execute = 'C:\Program Files (x86)\tidy-5.1.25-win32\bin\tidy'
let g:syntastic_shell='C:\Windows\System32\cmd'
" check debugging messages in vim with :mes:
let g:syntastic_debug = 1
let g:syntastic_mode_map = {"mode": "active",
\"active_filetypes" : ["html","javascript","json"],
\"passive_filetypes" : ["html", "javascript","json"] }
When I run
:SyntasticInfo
I get the following output
Syntastic version: 3.7.0-76 (Vim 704, Windows)
Info for filetype: html
Global mode: active
Filetype html is active
The current file will be checked automatically
Available checkers: -
Currently enabled checkers: -
For some reason, my default checkers aren't loading, and when I run
echo syntastic#util#system('echo %PATH%')
I get a E484 error
Error detected while processing function syntastic#util#system:
line 9:
E484: Can't open file C:\Users\Dirk\AppData\Local\Temp\VIo8366.tmp
-1
I suspect there are multiple issues here, so I assume the first one I need to solve is the E484 error. Any help appreciated.
I have a Flume consolidator that writes an entry from a custom log to s3 bucket in AWS
the problem i am having is, it is not storing it in JSON format. I am using flume-ng (flume 1.2.0) as i have upgraded from flume-og (really just flume 0.9.4-cdh3u3). When i was using flume (og one), i had it default to moving logs in JSON format without any params set. Is it possible for flume-ng to parse log and set it to JSON format?
Any help is much appreciated. Thank you
my setup config is below
agent.sources = source1
agent.sinks = sink1
agent.channels = channel1
agent.sources.source1.type = netcat
agent.sources.source1.bind = localhost
agent.sources.source1.port = 4555
agent.sinks.sink1.type=hdfs
agent.sinks.sink1.hdfs.path = s3://KEY:SECRET#BUCKET/flume/apache/incoming
agent.sinks.sink1.hdfs.filePrefix = log-file-
agent.channels.channel1.type = memory
agent.channels.channel1.capacity = 1000
agent.channels.channel1.transactionCapactiy = 100
agent.sources.source1.channels = channel1
agent.sinks.sink1.channel = channel1
I want to learn yii as my first framework. And I'm trying to make the contact form work. But I got this error:
I've already configured php.ini file from:
C:\wamp\bin\php\php5.3.0
And changed the default to these values:
[mail function]
; For Win32 only.
; http://php.net/smtp
SMTP = ssl:smtp.gmail.com
; http://php.net/smtp-port
smtp_port = 23
; For Win32 only.
; http://php.net/sendmail-from
sendmail_from = myemail#gmail.com
I've seen from here that gmail doesn't use port 25, which is the default in the php.ini. So I used 23. And also opened that port in the windows 7 firewall. Via inbound rules.
Then I also edited the main config in my yii application, to match the email that I'm using:
// application-level parameters that can be accessed
// using Yii::app()->params['paramName']
'params'=>array(
// this is used in contact page
'adminEmail'=>'myemail#gmail.com',
),
);
Finally, I restarted wampserver. Then cleared all my browsing data. Why then to I still see that its pointing out port 25 in the error. Have I miss something? Please help.
Heres a simple python script which could allow you to run a mail server on localhost, you dont have to change anything. Sorry if im a bit late.
import smtpd
import smtplib
import asyncore
class SMTPServer(smtpd.SMTPServer):
def __init__(*args, **kwargs):
print "Running fake smtp server on port 25"
smtpd.SMTPServer.__init__(*args, **kwargs)
def process_message(*args, **kwargs):
to = args[3][0]
msg = args[4]
gmail_user = 'yourgmailhere'
gmail_pwd = 'yourgmailpassword'
smtpserver = smtplib.SMTP("smtp.gmail.com",587)
smtpserver.ehlo()
smtpserver.starttls()
smtpserver.ehlo
smtpserver.login(gmail_user, gmail_pwd)
smtpserver.sendmail(gmail_user, to, msg)
print 'sent to '+to
pass
if __name__ == "__main__":
smtp_server = SMTPServer(('localhost', 25), None)
try:
asyncore.loop()
except KeyboardInterrupt:
smtp_server.close()
#end of code
Note: I used args[3][0] and args[4] as to address and message as the args sent by my php mail() corresponded to an array of args[3][0] as receipent email
If you open the php.ini file in WAMP, you will find these two lines:
smtp_server
smtp_port
Add the server and port number for your host (you may need to contact them for details)
The following two lines don't exist by default:
auth_username
auth_password
So you will need to add them to be able to send mail from a server that requires authentication. So an example may be:
smtp_server = mail.example.com
smtp_port = 25
auth_username = example_username#example.com
auth_password = example_password
ps: you should not use your personal mail here. for an obvious reason.
If using WAMP, the php.ini to be configured is present in the wamp/bin/apache/Apache_x_y/bin folder
where _x_y is related to the version of the Apache build used by your wamp installation
uncomment extension=php_openssl.dll at php.ini in WAMP server ("D:\wamp\bin\apache\Apache2.4.4\bin\php.ini")
In the file "D:\wamp\www\mantisbt-1.2.15\config_inc.php"
# --- Email Configuration ---
$g_phpMailer_method = PHPMAILER_METHOD_SMTP;
$g_smtp_host = 'smtp.gmail.com';
$g_smtp_connection_mode = 'ssl';
$g_smtp_port = 465;
$g_smtp_username = 'yourmail#gmail.com';
$g_smtp_password = 'yourpwd';
$g_enable_email_notification = ON;
$g_log_level = LOG_EMAIL | LOG_EMAIL_RECIPIENT;
$g_log_destination = 'file:/tmp/log/mantisbt.log';
$g_administrator_email = 'administrator#example.com';
$g_webmaster_email = 'webmaster#example.com';
$g_from_email = 'noreply#example.com';
$g_return_path_email = 'admin#example.com';
$g_from_name = 'Mantis Bug Tracker';
$g_email_receive_own = OFF;
$g_email_send_using_cronjob = OFF;