Can I split a single scrape target (a load balancer) into multiple targets, based on a label value? - amazon-elastic-beanstalk

Question

I have a Spring Boot application running on AWS Elastic Beanstalk. There are multiple application instances running. The number of running applications might dynamically increase and decrease from 2 to 10 instances.
I have set up Prometheus, but scraping the metrics has a critical limitation: it is only able to scrape the Elastic Beanstalk load balancer. This means that every scrape will return a different instance (round robin), so the metrics fluctuate wildly.
# prometheus.yml
scrape_configs:
- job_name: "my-backend"
metrics_path: "/metrics/prometheus"
scrape_interval: 5s
static_configs:
- targets: [ "dev.my-app.website.com" ] # this is a load balancer
labels:
application: "my-backend"
env: "dev"
(I am pursuing a correct set up, where Prometheus can directly scrape from the instances, but because of business limitations this is not possible - so I would like a workaround.)
As a workaround I have added a random UUID label to each application instance using RandomValuePropertySource
# application.yml
management:
endpoints:
enabled-by-default: false
web:
exposure:
include: "*"
endpoint:
prometheus.enabled: true
metrics:
tags:
instance_id: "${random.uuid}" # used to differentiate instances behind load-balancer
This means that the metrics can be uniquely identified, so on one refresh I might get
process_uptime_seconds{instance_id="6fb3de0f-7fef-4de2-aca9-46bc80a6ed27",} 81344.727
While on the next I could get
process_uptime_seconds{instance_id="2ef5faad-6e9e-4fc0-b766-c24103e111a9",} 81231.112
Generally this is fine, and helps for most metrics, but it is clear that Prometheus gets confused and doesn't store the two results separately. This is a particular problem for 'counters', as they are supposed to only increase, but because the different instances handle different requests, the counter might increase or decrease. Graphs end up jagged and disconnected.
I've tried relabelling the instance label (I think that's how Prometheus decides how to store the data separately?), but this doesn't seem to have any effect
# prometheus.yml
scrape_configs:
- job_name: "my-backend"
metrics_path: "/metrics/prometheus"
scrape_interval: 5s
static_configs:
- targets: [ "dev.my-app.website.com" ] # this is a load balancer
labels:
application: "my-backend"
env: "dev"
metric_relabel_configs:
- target_label: instance
source_labels: [__address__, instance_id]
separator: ";"
action: replace
To re-iterate: I know this is not ideal, and the correct solution is to directly connect - that is in motion and will happen eventually. For now, I'm just trying to improve my workaround, so I can get something working sooner.