Chrome preview differs from download - html

Question

I inspect the following page:
https://www.dm-jobs.com/Germany/search/?searchby=location&createNewAlert=false&q=&locationsearch=&geolocation=&optionsFacetsDD_customfield4=&optionsFacetsDD_customfield3=&optionsFacetsDD_customfield2=
or
https://www.dm-jobs.com/Germany/search/?q=&sortColumn=referencedate&sortDirection=desc&searchby=location&d=15.
As far as i understood the data can be either get via a get/post, in the "raw" html source or that some JavaScript code is executed.
But on that page i somehow dont manage to find the source.
The data on Chrome Network indicates that the data (here the Job data on the page) are in a Doc(ument) [see the screenshot - Tab Doc] and when i look on the preview tab its empty. But if i look on the "Response" tab the data can be seen.
Desired Output:
Target langauge is R, but actually not that relevant here. I would be happy enough to understand how the data is generated. So some selenium Approach or similar is not desired. But more getting an understanding how the data is generated and how it could be extracted via post/get, JS or the raw source.
What i tried:
library(httr)
library(rvest)
url <- "https://www.dm-jobs.com/Germany/search/?searchby=location&createNewAlert=false&q=&locationsearch=&geolocation=&optionsFacetsDD_customfield4=&optionsFacetsDD_customfield3=&optionsFacetsDD_customfield2="
src <- read_html(url)
src %>% html_nodes(xpath = "//*[contains(text(), 'Filialmitarbeiter')]")
as.character(src) %>% grep(pattern = "Filialmitarbeiter")
get <- GET(url)
content(get)
content(get$content)
Target Outputs:
e.g.
Filialmitarbeiter (w/m/d) 15-30 Std./Wo. Bad Reichenhall, DE, 83435 30.08.2019
Filialmitarbeiter (w/m/d) 6-8 Std./Wo. Neuenburg am Rhein, DE, 79395 30.08.2019
Führungsnachwuchs Filialleitung (w/m/d) Vechta, DE, 49377 30.08.2019

Answer 1

There are two cookies that are of import that must be picked up from the initial landing page. You can use html_session to capture these dynamically and then pass them on in a subsequent request to the page you want results from (at least for me). I wrote some stuff about session objects here.
The 3 cookies seen are:
cookies = c(
'rmk12' = '1',
'JSESSIONID' = 'some_value',
'cookie_j2w' = 'some_other_value'
)
You can find these plus the headers by using the network tab to monitor the web-traffic when attempting to view the job listings.
You can experiment with removing headers and cookies and you will discover that only the second and third cookies are required and no headers. However, the cookies passed must be captured in a prior request to the url as shown below. Session is the traditional way to do this.
R
library(rvest)
library(magrittr)
start_link = 'https://www.dm-jobs.com/Germany/?locale=de_DE'
next_link <- 'https://www.dm-jobs.com/Germany/search/?searchby=location&createNewAlert=false&q=&locationsearch=&geolocation=&optionsFacetsDD_customfield4=&optionsFacetsDD_customfield3=&optionsFacetsDD_customfield2='
jobs <- html_session(start_link) %>%
jump_to(.,next_link) %>%
html_nodes('.jobTitle-link') %>%
html_text()
print(jobs)
Py
import requests
from bs4 import BeautifulSoup as bs
with requests.Session() as s:
r = s.get('https://www.dm-jobs.com/Germany/?locale=de_DE')
cookies = s.cookies.get_dict() # just to demo which cookies are captured
print(cookies) # just to demo which cookies are captured
r = s.get('https://www.dm-jobs.com/Germany/search/?searchby=location&createNewAlert=false&q=&locationsearch=&geolocation=&optionsFacetsDD_customfield4=&optionsFacetsDD_customfield3=&optionsFacetsDD_customfield2=')
soup = bs(r.content, 'lxml')
print(len(soup.select('.jobTitle-link')))
Reading:
html_session