Return null values in conditional join - mysql

Question

I have a settings table and a settings_values table that cross matches the value for every user. It's important that I return a NULL value in cases where the setting has not been turned on for a user. It's also important that the table is highly efficient as it will be called often.
Ultimately, I have about 50 settings and thousands of users, so every time I run the query I should get 50 result similar to this:
setting_id | setting_name | value | user_id
1 blue true 35
2 red NULL NULL
3 yellow false 35
4 brown true 35
5 black NULL NULL
I have a working solution below that's running, but it's creating a bottleneck in my database requests which is strange given that settings only has 50 rows and settings_values only has about 20,000.
My tables are structured similar to this:
settings:
setting_id | setting_name
With setting_id as the only index (Primary Key)
settings_values:
id | setting_id | value | user_id
With id being the Primary Key, setting_id being a foregin key reference to the previous table, value is varchar and user_id also has an index on it.
My current working code is the following:
SELECT
s.setting_id,
s.setting_name,
sv.value,
sv.user_id
FROM
cmssps_settings s
LEFT JOIN cmssps_settings_values sv ON
s.setting_id = sg.setting_id
AND sv.user_id = '35';
That seems to work OK apart from the bottleneck (I get my 5 rows returned), and I suspect its the JOIN ON AND that's causing the slow down. I've been trying to achieve the same result with a JOIN ON WHERE like this, but I don't get my required NULL results (i.e. it only returns the 3 rows that satisfy the WHERE):
SELECT
s.setting_id,
s.setting_name,
sv.value,
sv.user_id
FROM
cmssps_settings s
LEFT JOIN cmssps_settings_values sv ON
s.setting_id = sg.setting_id
WHERE sv.user_id = '35';
When I EXPLAIN both I get this for the former:
id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
1 | SIMPLE | s | ALL | [NULL] | [NULL]| [NULL] | [NULL]| 49 | [NULL]
1 | SIMPLE | sv | ref | user_id | branch_id | 5 | const | 24 | Using where
When I do it for the latter I get a key value for my first table which I assume indicates that it will excecute faster:
id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
1 | SIMPLE | s | index | [NULL] | PRIMARY | 4 | [NULL]| 49 | [NULL]
1 | SIMPLE | sv | ref | user_id | branch_id | 5 | const | 24 | Using where
I've also attempted nesting the queries and using UNION but that seems like a regression.
Is there anything obvious as to why my initial approach is having such a negative impact on performance?
Is there a more efficient way to achieve the same outcome?