How can I set memory in hadoop-env.sh?
# Extra Java runtime options
# export HADOOP_OPTS=-server
If I want to set NN:20G DN:4G, what can I do?
You can set them in hadoop-env.sh.
export HADOOP_NAMENODE_OPTS="-Xmx20G ... "
export HADOOP_DATANODE_OPTS="-Xmx4G ... "
Related
myFile.asc is a large point cloud with million of points (rows) in the form of
-13.204000000 30.858300000 15.898300000
-13.204300000 30.864600000 15.899400000
-13.195000000 30.857300000 15.901000000
P = dlmread('/myPath/myFile.asc', " ")
stores the pointcloud successfully in P but unfortunately gives each line to console out in Octave. How can I suppress the logging here?
You can add a semicolon at the end to suppress the output:
P = dlmread('/myPath/myFile.asc', ' ');
My application runs well on RHEL 5 (TCL 8.4). But in RHEL 7 64bit TCL8.5, the database files are not sourced in correctly. The application is by default pointing to the last record file in the db. Hence, im assuming it might be issue with the way 8.5 handles the file sourcing. So i created a file X and wrote the below code. (Please ignore dbname and /path, it works fine)
File X
#!/bin/sh
# \
exec tclsh "$0" "$#"
puts [package require Itcl]
namespace import ::itcl::*
puts $itcl::patchLevel
puts $itcl::library
set databases /dbname
set system ($databases,dbpath) /path
source File.class.tcl
source FareFile.class.tcl
puts [Fare.File formtitle]
source Record1File.class.tcl
puts [Fare.File formtitle]
I source FareFile in and print the form title(o/p: Fare Viewer) using the formtitle method which is declared in the File.class.tcl. And then I source Record1File and print the FareFile formtitle (the first one), its printing the form title of Record1File. The formtitle method is returning the value of the lastest sourced file. This does not happen in 8.4
File.class.tcl:
class File {
variable fileinfo
variable recordarray
variable allads_flag 0
variable updates_is_lastkey 0
method formtitle {} {
variable fileinfo
return $fileinfo(formtitle)
}
}
FareFile.class.tcl
FareFile ::Fare.File
::Fare.File parse_fields {
tabtitle "Fares"
formtitle "Fare Viewer"
}
Record1File.class.tcl
Record1File ::Record1.File
::Record1.File parse_fields {
tabtitle "Record 1"
formtitle "Record 1 Viewer"
output in 8.4 / RHEL 5: (Expected output in 8.5)
3.4
3.4.0
/path
Fare Viewer
Fare Viewer
output in 8.5 / RHEL 7:
3.4
3.4.3
/path
Fare Viewer
Record 1 Viewer
If you see the output on both platforms, its different. Please help
(This is only a tentative answer, given all the blanks of the question, but I need the formatting capabilities:)
Try the following pls., by rewriting the body script of method formtitle as follows:
class File {
method formtitle {} {
set v variable
$v fileinfo
return $fileinfo(formtitle)
}
}
... and report back by posting a comment.
I'm running gunicorn as:
guiconrn --bind=0.0.0.0:5000 --log-file gunicorn.log myapp:app
Seems like gunicorn.log keeps growing. Is there a way to specify a max size of the log file, so that if it reaches max size, it'll just override it.
Thanks!!
TLDR;
I believe there might be a "python only" solution using the rotating file handler provided in the internal lib of python. (at least 3.10)
To test
I created a pet project for you to fiddle with:
Create the following python file
test_logs.py
import logging
import logging.config
import time
logging.config.fileConfig(fname='log.conf', disable_existing_loggers=False)
while True:
time.sleep(0.5)
logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
Create the following config file
log.conf
[loggers]
keys=root
[handlers]
keys=rotatingHandler
[formatters]
keys=sampleFormatter
[logger_root]
level=DEBUG
handlers=rotatingHandler
[handler_rotatingHandler]
class=logging.handlers.RotatingFileHandler
level=DEBUG
formatter=sampleFormatter
args=('./logs/logs.log', 'a', 1200, 1, 'utf-8')
[formatter_sampleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
Create the ./logs directory
Run python test_logs.py
To Understand
As you may have noticed already, the setting that allow for this behaviour is logging.handlers.RotatingFileHandler and the provided arguments args=('./logs/logs.log', 'a', 1200, 10, 'utf-8')
RotatingFileHandler is a stream handler writing to a file. That allow for 2 parameters of interest:
maxBytes set arbitrarily at 1200
backupCount set arbitrarily to 10
The behaviour is that upon reaching 1200 Bytes in size, the file is closed, renamed to /logs/logs.log.<a number up to 10> and a new file is opened.
BUT is any of maxBytes or backupCount is 0. No rotation is done !
In Gunicorn
As per the documentation you can feed a config file.
This could look like:
guiconrn --bind=0.0.0.0:5000 --log-config log.conf myapp:app
You will need to tweak it to your existing setup.
On Ubuntu/Linux, suggest to use logrotate to manage your logs, do like this: https://stackoverflow.com/a/55643449/6705684
Since Python>3.3, With RotatingFileHandler, here is my solution(MacOS/Windows/Linux/...) :
import os
import logging
from logging.handlers import RotatingFileHandler
fmt_str = '[%(asctime)s]%(module)s - %(funcName)s - %(message)s'
fmt = logging.Formatter(fmt_str)
def rotating_logger(name, fmt=fmt,
level=logging.INFO,
logfile='.log',
maxBytes=10 * 1024 * 1024,
backupCount=5,
**kwargs
):
logger = logging.getLogger(name)
hdl = RotatingFileHandler(logfile, maxBytes=maxBytes, backupCount=backupCount)
hdl.setLevel(level)
hdl.setFormatter(fmt)
logger.addHandler(hdl)
return logger
more refer:
https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler
import os, sys
AWS_DIRECTORY = '/home/jenkins/.aws'
certificates_folder = 'my_folder'
SUCCESS = 'success'
class AmazonKMS(object):
def __init__(self):
# making sure boto3 has the certificates and region files
result = os.system('mkdir -p ' + AWS_DIRECTORY)
self._check_os_result(result)
result = os.system('cp ' + certificates_folder + 'kms_config ' + AWS_DIRECTORY + '/config')
self._check_os_result(result)
result = os.system('cp ' + certificates_folder + 'kms_credentials ' + AWS_DIRECTORY + '/credentials')
self._check_os_result(result)
# boto3 is the amazon client package
import boto3
self.kms_client = boto3.client('kms', region_name='us-east-1')
self.global_key_alias = 'alias/global'
self.global_key_id = None
def _check_os_result(self, result):
if result != 0 and raise_on_copy_error:
raise FAILED_COPY
def decrypt_text(self, encrypted_text):
response = self.kms_client.decrypt(
CiphertextBlob = encrypted_text
)
return response['Plaintext']
when using it
amazon_kms = AmazonKMS()
amazon_kms.decrypt_text(blob_password)
getting
E ClientError: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
stacktrace is
../keys_management/amazon_kms.py:77: in decrypt_text
CiphertextBlob = encrypted_text
/home/jenkins/.virtualenvs/global_tests/local/lib/python2.7/site-packages/botocore/client.py:253: in _api_call
return self._make_api_call(operation_name, kwargs)
/home/jenkins/.virtualenvs/global_tests/local/lib/python2.7/site-packages/botocore/client.py:557: in _make_api_call
raise error_class(parsed_response, operation_name)
This happens in a script that runs once an hour.
it's only failing 2 -3 times a day.
after a retry it succeed.
Tried to upgraded from boto3 1.2.3 to 1.4.4
what is the possible cause for this behavior ?
My guess is that the issue is not in anything you described here.
Most likely the login-tokes time out or something along those lines. To investigate this further a closer look on the way the login works here is probably helpful.
How does this code run? Is it running inside AWS like on Lambda or EC2? Do you run it from your own server (looks like it runs on jenkins)? How is the login access established? What are those kms_credentials used for and how do they look like? Do you do something like assumeing a role (which would probably work through access tokens which after some time will no longer work)?
I am struggling to create a simple rake task which will generate a csv dump of the database table "baselines".
task :send_report => :environment do
path = "tmp/"
filename = 'data_' + Date.today.to_s + '.csv'
Baseline.all.each do
CSV.open(path + filename, "wb") do |csv|
csv << Baseline.column_names
Baseline.all.each do |p|
csv << p.attributes.values_at(*column_names)
end
end
end
end
I am getting the error
undefined local variable or method `column_names' for main:Object
I am completely unclear why this is....Baseline.column_names will work in the console, in a view etc etc.
Any thought would be appreciated.
You're specifying Baseline.column_names in the first case, but just column_names on your values_at call. That defaults to the main context where no such method exists. It must be called against a model.
Make those two consistent, Baseline is required in both cases.