How to enable additional configs for prometheus operator - configuration

Question

according to Prometheus-operator documentation we should be able to supply our additional configuration easily via secret file. Does anybody actually succeed this step? I have several questions:
where these configuration will appear in prometheus pod?
should this configuration be in a form of prometheus configuration file or just list additional scrape entries
can we supply additional files (json configs) via file_sd_configs: and if so how to supply those files into prometheus manifest file?
Regardless of those questions I have hard time to add the additional configuration. I basically followed exact steps from documentation, and here is my observations:
here is my new configuration
cat prometheus-additional.yaml
- job_name: "prometheus-custom"
static_configs:
- targets: ["localhost:9090"]
add new secret file
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml
create prometheus.yml file with additional configuration
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
replicas: 2
resources:
requests:
memory: 400Mi
additionalScrapeConfigs:
name: additional-scrape-configs
key: prometheus-additional.yaml
deploy prometheus.yaml
kubectl apply -f prometheus.yaml
check logs and there is no indication of my new configuration
kubectl logs prometheus-prometheus-0 -c prometheus
level=info ts=2019-12-05T18:07:30.217852541Z caller=main.go:302 msg="Starting Prometheus" version=" (version=2.7.1, branch=HEAD, revision=62e591f928ddf6b3468308b7ac1de1c63aa7fcf3)"
level=info ts=2019-12-05T18:07:30.217916972Z caller=main.go:303 build_context="(go=go1.11.5, user=root#f9f82868fc43, date=20190131-11:16:59)"
level=info ts=2019-12-05T18:07:30.217971648Z caller=main.go:304 host_details="(Linux 4.19.3-300.fc29.x86_64 #1 SMP Wed Nov 21 15:27:25 UTC 2018 x86_64 prometheus-prometheus-0 (none))"
level=info ts=2019-12-05T18:07:30.217994128Z caller=main.go:305 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-12-05T18:07:30.218236509Z caller=main.go:306 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-12-05T18:07:30.219359123Z caller=main.go:620 msg="Starting TSDB ..."
level=info ts=2019-12-05T18:07:30.219487263Z caller=web.go:416 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-12-05T18:07:30.230944675Z caller=main.go:635 msg="TSDB started"
level=info ts=2019-12-05T18:07:30.231037536Z caller=main.go:695 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2019-12-05T18:07:30.23125837Z caller=main.go:722 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2019-12-05T18:07:30.231294106Z caller=main.go:589 msg="Server is ready to receive web requests."
level=info ts=2019-12-05T18:07:33.568068248Z caller=main.go:695 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2019-12-05T18:07:33.568305994Z caller=main.go:722 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
And, when I logged into prometheus pod I don't see any additional configuration either, and when I check my prometheus web console I don't see any of my configurations.

Answer 1

Turns out, the prometheus-operator still relies on serviceMonitorSelector: {} part of the manifest file according to this ticket. Therefore in order to add additional configuration we need to have the following manifest:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
replicas: 2
resources:
requests:
memory: 400Mi
additionalScrapeConfigs:
name: prometheus-config
key: prometheus-config.yaml
serviceMonitorSelector: {}
where prometheus-config.yaml will contain prometheus scrape rules and deployed via secrets to prometheus cluster. I also found empirically that current prometheus-operator does not support file_sd_configs in prometheus configuration (sad) and someone need to write full rules in prometheus-config.yaml file.

Answer 2

The documentation is somehow wrong, a current operator does not need the serviceMonitorSelector, but using the above configuration throws a
cannot unmarshal !!map into []yaml.MapSlice error.
The correct reference is additionalScrapeConfigsSecret:
additionalScrapeConfigsSecret:
enabled: true
name: additional-scrape-configs
key: prometheus-additional.yaml
Else you get the error cannot unmarshal !!map into []yaml.MapSlice
Here is a better documentation:
https://github.com/prometheus-community/helm-charts/blob/8b45bdbdabd9b54766c4beb3c562b766b268a034/charts/kube-prometheus-stack/values.yaml#L2691
According to this, you could add scrape configs without packaging into a secret like this:
additionalScrapeConfigs: |
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]