I had been trying to up the memory of my elastic beanstalk console using JAVA_OPTS in environment settings with values -Xms1G -Xmx3G. Attached is the image on how I have changed the settings.
AFter applying the changes and restarting the vm, I do not see the changes refelcted on the server.
This is how I am verifying
sudo jmap -heap
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 1035993088 (988.0MB)
NewSize = 21495808 (20.5MB)
MaxNewSize = 344981504 (329.0MB)
OldSize = 43515904 (41.5MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 192413696 (183.5MB)
used = 18710296 (17.843528747558594MB)
free = 173703400 (165.6564712524414MB)
9.723993867879342% used
From Space:
capacity = 26738688 (25.5MB)
used = 22166296 (21.139427185058594MB)
free = 4572392 (4.360572814941406MB)
82.89971445121017% used
To Space:
capacity = 27262976 (26.0MB)
used = 0 (0.0MB)
free = 27262976 (26.0MB)
0.0% used
PS Old Generation
capacity = 691011584 (659.0MB)
used = 571332904 (544.8655166625977MB)
free = 119678680 (114.13448333740234MB)
Heap settings cannot be set through environment properties. You have to give this via Procfile. The procfile has tobe bundled when uplaoding.
I had to created a zip file that had war and Procfile.
Proc file contents
web: java -jar -Xms1G -Xmx3G application.war
How to test this works?
Find the process id of your webapp/java process from top.
Use jmap heap - to get the heap allocation. I tested this on AWS-Ec2 for elastic beanstalk
Related
I'm monitoring about 1300 hosts on Zabbix. After I defined bulk templates for all hosts, I got the "Utilization of housekeeper processes over 75%" alarm, but it was not resolved for about 20 hours. I did not define a housekeeper in my server settings. How can I resolve this alarm and what is the effect? Using postgresql.
Server config;
StartPollers=50
StartPollersUnreachable=50
StartPingers=50
StartDiscoverers=50
StartHTTPPollers=50
CacheSize=1024M
HistoryCacheSize=1024M
TrendCacheSize=1024M
ValueCacheSize=1024M
LogSlowQueries=3000
MaxHousekeeperDelete=5000
Postgreqsql config;
max_connections = 1000
shared_buffers = 8GB
effective_cache_size = 24GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 1048kB
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 12
max_parallel_workers_per_gather = 4
max_parallel_workers = 12
max_parallel_maintenance_workers = 4
Such a problem, there is a server (cluster) on which smb is used, the server is entered into the AD domain, sometimes it is necessary to restart the smbd service (reload won't fit), but at the same time there is some copying of the file on the client (windows), then the load is interrupted, and after the klick "Retry" button, the download starts from the very beginning. Is it possible to do something like that so that the load continues to go from the moment where it was interrupted, maybe you need to configure the client like that. client connects as SMBv3 or SMBv2
server on ubuntu 18.04.
smb created at zfs
smb.conf:
[global]
workgroup = TEST247
realm = test247.ru
security = ads
auth methods = winbind
interfaces = 172.16.11.170/24
bind interfaces only = yes
netbios name = SERVER
encrypt passwords = true
map to guest = Bad User
max log size = 300
dns proxy = no
socket options = TCP_NODELAY
domain master = no
local master = no
preferred master = no
os level = 0
domain logons = no
load printers = no
show add printer wizard = no
log level = 0 vfs:2
max log size = 0
syslog = 0
printcap name = /dev/null
disable spoolss = yes
name resolve order = lmhosts wins host bcast
machine password timeout = 604800
name cache timeout = 660
idmap config TEST247 : backend = rid
idmap config TEST247 : base_rid = 0
idmap config TEST247 : range = 100000 - 200000
idmap config * : range = 200001-300000
idmap config * : backend = tdb
idmap cache time = 604800
idmap negative cache time = 60
winbind rpc only = yes
winbind cache time = 120
winbind enum groups = yes
winbind enum users = yes
winbind max domain connections = 10
winbind use default domain = yes
winbind refresh tickets = yes
winbind reconnect delay = 15
winbind request timeout = 25
winbind separator = ^
private dir = /var/lib/samba/private
lock directory = /run/samba
state directory = /var/lib/samba
cache directory = /var/cache/samba
pid directory = /run/samba
log file = /var/log/samba/smb.%m
include = /etc/samba/smb-res.conf
testparm:
testparm -s /etc/samba/smb.conf
Load smb config files from /etc/samba/smb.conf
WARNING: The "auth methods" option is deprecated
WARNING: The "syslog" option is deprecated
Loaded services file OK.
Server role: ROLE_DOMAIN_MEMBER
smb-res.conf:
[test109_smb]
comment = test109_smb share
path = /config/pool/test109/smb
browseable = yes
writable = yes
inherit acls = yes
inherit owner = no
inherit permissions = yes
map acl inherit = yes
nt acl support = yes
create mask = 0777
force create mode = 0777
force directory mode = 0777
store dos attributes = yes
public = no
admin users =
valid users =
write list =
read list =
invalid users =
vfs objects = acl_xattr
full_audit:prefix = %S|%u|%I
full_audit:facility = local5
full_audit:priority = notice
full_audit:success = none
full_audit:failure = none
shadow: snapdir = .zfs/snapshot
shadow: sort = desc
shadow: localtime = yes
shadow: format = shadow_%d.%m.%Y-%H:%M:%S
worm: grace_period = 30
cryptfile: method = grasshopper
Resuming a copy operation doesn't depend on the smb client or server, but on the application which is doing the copying.
The standard Windows copy doesn't know to resume.
Other (third party) apps (maybe Total Commander?) can be more intelligent about it. You could even write your own app to do a smart copy.
I am running a simple comment classification task on google colab. I am using DistilBERT for contextual embeddings.I use only 4000 training sample cause the notebook keeps on crashing.
When I run the cell for obtaining the embeddings, I keep a tab on how the RAM utilisation increases. I am seeing that it oscillates from somewhere between 3gb to 8gb.
Should not it be just increasing? Can anyone explain how this works at lower level.
Here is my code, the cell block at last is where I am seeing the above said thing.
# For DistilBERT:
model_class, tokenizer_class, pretrained_weights = (ppb.DistilBertModel, ppb.DistilBertTokenizer, 'distilbert-base-uncased')
## Want BERT instead of distilBERT? Uncomment the following line:
#model_class, tokenizer_class, pretrained_weights = (ppb.BertModel, ppb.BertTokenizer, 'bert-base-uncased')
# Load pretrained model/tokenizer
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)
max_len=80
tokenized = sample['comment_text'].apply((lambda x: tokenizer.encode(x, add_special_tokens=True,max_length= max_len)))
padded = np.array([i + [0]*(max_len-len(i)) for i in tokenized.values])
attention_mask = np.where(padded != 0, 1, 0)
attention_mask.shape
input_ids = torch.tensor(padded)
attention_mask = torch.tensor(attention_mask)
**with torch.no_grad():
last_hidden_states = model(input_ids, attention_mask=attention_mask)**
I have updated the logging.properties file of tomcat to print the logs in json format.
But, the issue is value of "message " has some escape characters, which makes my log invalid json.
Please let me know how to escape these characters(:,[,],/) in json using default tomcat-juli.jar and used as a string.
Below is my updated logging.properties file:
handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, 3manager.org.apache.juli.AsyncFileHandler, 4host-manager.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler
.handlers = 1catalina.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################
java.util.logging.SimpleFormatter.format = {"date" :"%1$tF", "timestamp": "%1$tT" , "loggerlevel": "%4$s", "loggersource": "%3$s" , "message": "%5$s%6$s"}%n
1catalina.org.apache.juli.AsyncFileHandler.level = FINE
1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
1catalina.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
2localhost.org.apache.juli.AsyncFileHandler.level = FINE
2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
2localhost.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
3manager.org.apache.juli.AsyncFileHandler.level = FINE
3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
3manager.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
4host-manager.org.apache.juli.AsyncFileHandler.level = FINE
4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
4host-manager.org.apache.juli.AsyncFileHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.AsyncFileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = 3manager.org.apache.juli.AsyncFileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers = 4host-manager.org.apache.juli.AsyncFileHandler
I would suggest you official way (without using middleware):
Download Elastic JUL formatter and logging core libraries:
https://mvnrepository.com/artifact/co.elastic.logging/jul-ecs-formatter
https://mvnrepository.com/artifact/co.elastic.logging/ecs-logging-core
Put them into tomcat/bin folder
List them in CLASSPATH system property in tomcat/bin/setenv.sh(bat).
If no such file - create it with the next content: on Unix -
export CLASSPATH="$CLASSPATH:ecs-logging-core.jar:jul-ecs-formatter.jar"
on Windows - set CLASSPATH=%CLASSPATH%;ecs-logging-core.jar;jul-ecs-formatter.jar
Edit tomcat/conf/logging.properties in such way:
handlers = java.util.logging.ConsoleHandler
.handlers = java.util.logging.ConsoleHandler
...
java.util.logging.ConsoleHandler.formatter = co.elastic.logging.jul.EcsFormatter
Such configuration will make tomcat to write all events in appropriate Elastic (Logstash) JSON format into console (stdout) which is very suitable when you running in docker or other container.
Documentation https://www.elastic.co/guide/en/ecs-logging/java/1.x/setup.html
I would do the following:
Download and build devatherock's jul json formatter
Download json-simple
Add both to your tomcat startup classpath in the $CATALINA_BASE/bin/setenv.sh script
In $CATALINA_BASE/conf/logging.properties, add or change the formatters for your log handlers to io.github.devatherock.json.formatter.JSONFormatter
We have Samba setup for our shared drive. I have pasted the smb.conf file below. Everything is working well accept when we try and run an EXE file using Windows Vista. When we run an EXE file it first ask for UAC control then it pops up the username and password prompt. You must then type your username and password in again before it will run.
I think the issues is that UAC is now running the application under Admin instead of the logged in user. So the first username and password that is cached is not seen by the admin user. Does anyone know of a work around for this?
smb.conf:
[global]
passdb backend = tdbsam
security = user
encrypt passwords = yes
preferred master = Yes
workgroup = Workgroup
netbios name = Omni
bind interfaces only = True
interfaces = lo eth2
;max disk size = 990000 ;some programs (like PS7) can't deal with more than 1TB
socket options = TCP_NODELAY
server string = Omni
;smb ports = 139
debuglevel = 1
syslog = 0
log level = 2
log file = /var/log/samba/%U.log
max log size = 61440
vfs objects = omnidrive recycle
recycle:repository = RecycleBin/%U
recycle:keeptree = Yes
recycle:touch = No
recycle:versions = Yes
recycle:maxsize = 0
recycle:exclude = *.temp *.mp3 *.cat
omnidrive:log = 2
omnidrive:com_log = 1
omnidrive:vscan = 1
omnidrive:versioningState = 1
omnidrive:versioningMaxFileSize = 0
omnidrive:versioningMaxRevSize = 7168
omnidrive:versioningMaxRevNum = 1000
omnidrive:versioningMinRevNum = 0
omnidrive:versioningfilesInclude = /*.doc/*.docx/*.xls/*.xlsx/*.txt/*.bmp/
omnidrive:versioningfilesExclude = /*.tmp/*.temp/*.exe/*.com/*.jarr/*.bat/.*/
full_audit:failure = none
full_audit:success = mkdir rename unlink rmdir write open close
full_audit:prefix = %u|%I|%m|%S
full_audit:priority = NOTICE
full_audit:facility = LOCAL6
;dont descend = RecycleBin
veto files = /.subversion/*.do/*.do/*.bar/*.cat/
client ntlmv2 auth = yes
[netlogon]
path = /var/lib/samba/netlogon
read only = yes
[homes]
read only = yes
browseable = no
[share1]
path = /share1
read only = no
browseable = yes
writable = yes
admin users = clinton1
public = no
create mask = 0770
directory mask = 0770
nt acl support = no
;acl map full control = no
hide unreadable = yes
store dos attributes = yes
map archive = no
map readonly = Permissions
If anyone cares; this is how I fixed the issues on vista:
I set a key to link the UAC account and the none UAC account.
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
EnableLinkedConnections =(dword)1
The password prompt goes away.
I think that you can also address this by turning off UAC in Vista or Windows 7. Here's a link for doing that: Turn User Account Control on or off