Mysql "memory usage" increasing and increasing - mysql

Question

I have a really big website built old fashioned with PHP & MYSQL.
I have more than 1,000 different queries in my website, on different PHP pages, and it's really hard to update all of them to MYSQLI.
I bought VPS server with 4GB RAM and in the past months I experience really slow page loads.
When I restart my server, everything runs smoothly, but after couple of hours/days the website is getting muchu slower with loading time of 3+ seconds for a page load. I notice that the mysqld service is increasing and increasing in memory usage, from 80MB on server restart it reached about 400MB and more of usage.
I put in the end of my index.php mysql_close() but it seem like the connection number still increasing.
Questions
What can cause unlimited increment in mysql memory usage?
Updating all my queries to MYSQLI may improve the performance?
Some information:
innodb_version
5.5.31
protocol_version
10
slave_type_conversions
version
5.5.31-log
version_comment
MySQL Community Server (GPL)
version_compile_machine
x86_64
version_compile_os
Linux
storage engine: Mixed (Somes tables are INNODB,some tables are MyISAM.
my.cnf:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
max-connections=100000
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
thread_cache_size=5
table_open_cache=99390
sort_buffer_size=512M
read_rnd_buffer_size=512M
query_cache_size=512M
query_cache_limit = 16M
query_cache_type = 1
slow_query_log=1
slow_query_log_file=slow_query_log.log #
long_query_time=5
log-queries-not-using-indexes=1
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
I have about ~6-7 queries running when I use show processlist

Answer 1

max-connections=100000 -- Yikes! Drop to 1000
table_open_cache=99390 -- drop to, say, 2000
sort_buffer_size=512M -- drop to 1% of RAM, say, 40M
read_rnd_buffer_size=512M -- ditto
query_cache_size=512M -- too big; slows things down; drop to 40M
long_query_time=5 -- not low enough to catch much; drop to 2
log-queries-not-using-indexes=1 -- clutters the slowlog without providing much info; change to 0
You did not say which Engine you are using. Read this for advice on MyISAM and InnoDB.
1000 pages -- that's not too many.
Which web server? If Apache, don't set MaxClients to more than 20.
2022 postscript: The query_cache_size and query_cache_type variables have been removed from mySQL 8.0.3+.