static div on scaled img - html

Question

Is it possible to adress only the image (id doesn't work because it includes the code between the brackets), to get scaled when hovered? Div + text should stay the same size.
I tried to scale the div + text in the other way round (1/1.1=0.9). But it wobbles, when the mouse leaves the area.
.center {
position: absolute;
}
.scrolldownbutton {
background-color: green;
position: relative; /* absolute um ueber Text zu schweben*/
width: 300px;
height: 300px;
border: 1px solid black;
text-align: center;
line-height: 300px;
transition: all .5s ease;
}
.scrolldownbutton:hover {
transform: scale(1.1);
}
.rahmen {
border: 2px solid yellow;
margin: 5%;
height: 90%;
}
.rahmen:hover {
transition: all .5s ease;
transform: scale(0.9);
}
.text {
margin: -10%;
font-size: 300%;
color: white;
}
#newYork {
background-image: url('http://www.travelersjoy.com/images/planning/destinations/new-york-city/newyork1.jpg');
}
<div class="center">
<div id="newYork" class="scrolldownbutton">
<div class="rahmen">
<p class="text">New York</p>
</div>
</div>
</div>