XPath expression fails to return correct elements in Selenium (Java) - selenium-chromedriver

I am trying to collect a list of products in Amazon. Specifically, I am going to the following URL: https://www.amazon.com/dp/[ASIN]/ref=olp-opf-redir?aod=1&ie=UTF8&condition=ALL where [ASIN] is the unique Amazon Standard Identification Number for the item in question. For this issue, assume the URL to be for these airpods: https://www.amazon.com/dp/B09JQMJHXY/ref=olp-opf-redir?aod=1&ie=UTF8&condition=ALL
Notice that this URL opens a side panel with a listing of different vendors selling the item in different conditions (i.e. new, used, used like new, etc.).
I created an XPath expression to obtain some of these items. The basic XPath for this is //div[#id='aod-offer-list']/div[#id='aod-offer']. I further refined this to return a list of items that are shipped only from Amazon:
//div[#id='aod-offer-list']/div[#id='aod-offer' and div[#id='aod-offer-shipsFrom']/div/div/div/span[text()='Amazon']]
When I evaluate this expression in Chrome, I get the list of offers I am interested in. However, when I run this from Eclipse, I get a list of offers that consist of multiple copies of the pinned offer at the very top of the side panel. The bizarre thing is that the pinned offer (//div[#id='aod-pinned-offer']) is not even a child of the offer list (//div[#id='aod-offer-list']). In fact, the pinned offer and the offer list are siblings of each other. Given these facts, how is it that I am getting a different WebElement list when executing in Java than when evaluating the same XPath directly in Chrome.
The relevant code:
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver", "C:/Program Files/WebDrivers/chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.get("https://www.amazon.com/dp/B09JQMJHXY/ref=olp-opf-redir?aod=1&ie=UTF8&condition=ALL");
List<WebElement> offers = new ArrayList<>();
try {
// merchants = driver.findElements(By.xpath(xpath));
new WebDriverWait(driver, Duration.ofSeconds(10)).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//div[#id='aod-offer-list']")));
String xpath = "//div[#id='aod-offer-list']/div[#id='aod-offer' and div[#id='aod-offer-shipsFrom']/div/div/div/span[text()='Amazon']]";
offers = new WebDriverWait(driver, Duration.ofSeconds(10)).until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.xpath(xpath)));
System.out.println("Found " + offers.size() + " offers.");
Iterator<WebElement> iter = offers.iterator();
while (iter.hasNext()) {
String script = "return arguments[0].innerHTML";
WebElement offer = iter.next();
WebElement soldByElement = offer.findElement(By.xpath("//a[#aria-label='Opens a new page']"));
String soldByText = (String) ((JavascriptExecutor) driver).executeScript(script, soldByElement);
System.out.println("Sold by: " + soldByText);
WebElement priceElement = offer.findElement(By.xpath("//span[#class='a-offscreen']"));
String priceString = (String) ((JavascriptExecutor) driver).executeScript(script, priceElement);
System.out.println("Price for item " + priceString);
}
} catch (TimeoutException toe) {
System.err.println(toe);
}
driver.quit();
}
The output:
Found 4 offers.
Sold by: Adorama
Price for item $174.00
Sold by: Adorama
Price for item $174.00
Sold by: Adorama
Price for item $174.00
Sold by: Adorama
Price for item $174.00
The output should've been:
Found 2 offers.
Sold by: Amazon Warehouse
Price for item $160.08
Sold by: Amazon Warehouse
Price for item $165.30
The incorrect output is pulling the price from the pinned item and the "Sold By" value from one of the vendors not shipping from Amazon. My unproven theory is that the relative path to the "Sold By" and "Price" elements are not relative from the offer element, but from the DOM itself. I tried adding a dot (.) to the XPath string, but that is not a correct notation. I need to force Selenium to resolve the path starting from the obtained offer element.
UPDATE:
If I add the following snippet
String script = "return arguments[0].innerHTML";
WebElement offer = iter.next();
String offerElement = (String) ((JavascriptExecutor) driver).executeScript(script, offer);
System.out.println(offerElement);
it prints out the correct "innerHTML" for the list of offers. In other words, I can see all the correct elements if I use this Xpath
String xpath = "//div[#id='aod-offer-list']/div[#id='aod-offer']";

Related

azure-devops export work items as csv and include ALL comments/discussions

if I make a query in Azure Devops for my work items, when I add columns to display, there is only comments count and discussions, but is there any where to include all comments from a work item in my output saved csv file? I need to archive the comments for every work item in a csv.
Can I do this through the azure desktop web ui somehow? or do i need to write my own script using the azure api to view comments and add them to a work item
Can I do this through the azure desktop web ui somehow? or do i need
to write my own script using the azure api to view comments and add
them to a work item
For your first question, the answer is NO. Since you want all the comments for a workitem, then built-in UI feature will not be able to achieve your requirement.
For your second the question, the answer is YES.
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
import requests
import csv
import os
#get all the comments of a work item
def get_work_items_comments(wi_id):
#get a connection to Azure DevOps
organization_url = 'https://dev.azure.com/xxx'
personal_access_token = 'xxx'
credentials = BasicAuthentication('', personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)
work_item_tracking_client = connection.clients.get_work_item_tracking_client()
#get the work item
work_item = work_item_tracking_client.get_work_item(wi_id)
#get the comments of the work item
comments_ref = work_item._links.additional_properties['workItemComments']['href']
#send a request to get the comments
response = requests.get(comments_ref, auth=('', personal_access_token))
#get the comments
comments = response.json()['comments']
return comments
#return work item id, work item title and related work item comments
def get_work_items_results(wi_id):
#get a connection to Azure DevOps
organization_url = 'https://dev.azure.com/xxx'
personal_access_token = 'xxx'
credentials = BasicAuthentication('', personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)
work_item_tracking_client = connection.clients.get_work_item_tracking_client()
#get the work item
work_item = work_item_tracking_client.get_work_item(wi_id)
#get the title of the work item
title = work_item.fields['System.Title']
#get the work item id
id = work_item.id
#get the comments of the work item
items = get_work_items_comments(wi_id)
array_string = []
for item in items:
text = item['text']
array_string.append(text)
print(item['text'])
return id, title, array_string
#Save the work item id, work item title and related work item comments to a csv file
#create folder workitemresults if not exist
if not os.path.exists('workitemresults'):
os.makedirs('workitemresults')
with open('workitemresults/comments_results.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(['Workitem ID','Workitem Title','Workitem Comments'])
#===if you want multiple work items, just for loop in this place and replace the value 120 in this place, 120 is the workitem id on my side.===
writer.writerow(get_work_items_results(120))
The above code is to capture the comments and information for one workitem. It works fine on my side(The place for what 'for loop' should be placed I had already mentioned in my code, using the for loop in that place you can capture multiple workitems):
In my situation, if I only want the text content:
#remove <div> and </div> from the text
text = text.replace('<div>','')
text = text.replace('</div>','')
Result:

CFML/JS Creating nested JSON/Array from plain SQL

I would like to build a tree structre from a plain json array.
The regular depth is approx. 6/7 (max 10) and has about 5,000 records.
My input json looks like this
[3,"01","GruppenAnfangHook",1,0,1,0,"Installationsmaterial",1.0,"",null,null,0.0,-1.0,null,803.0300,803.0300,0.00000,1,1]
[5,"01.001","JumboAnfangHook",3,0,3,0,"MBS Wandler 1.000",6.0,"St",null,null,0.0,-6.0,0.0000,336.7800,56.1300,0.00000,2,2],
[38,"","ArtikelHook",3,5,3,0,"ASK 61.4 1000/5A 5VA Kl.1 Preis lt. Hr. K am 16.05.17",6.0,"stk",6.0,6.0,0.0,-6.0,null,21.5000,21.5000,0.00000,3,3]
But I need it structured with childrens like that
{"0":34,1":"02.003",2":"JumboBegin","3":26,"4":0, "5":26,"6":0, "children":[
{ "0":36,"1":"", "2":"Article","3":26,"4":34,"5":26,6:"0", 7: "Artikel"},
{ "0":35,"1":"", "2":"JumboEnd",3":26,"4":34, "5":26, 6:"0",7:"Stunde"}
]}
My best approach so far was to build the child-structure with the following JS function in the frontend
function nest(data, parentId = 0) {
return data.reduce((r, e) => {
let obj = Object.assign({}, e)
if (parentId == e[4]) {
let children = nest(data, e[0])
if (children.length) obj.children = children
r.push(obj)
}
return r;
}, [])}
It works well and fast (< 1s) with a small (<500) amount of records but my browser begins to freeze at 2,000 and above.
My thought was it is too much data and so I tried to solve it in the CFML backend.
Due to I'm new with recursion, Ben Nadels Blog helped me alot, so I used his post about recursion and created a working example with sample data.
q = queryNew("id,grpCol,jumCol,leiCol,name,typ,order");
The grpCol is level 0, up to 5 groups can be placed in each other, in those groups can be placed two kinds of containers (jumCol and leiCol), they can be placed in each other to, but not in themselfs.
But now I am failing to convert it to a array of structures with child members. The structure of the HTML tree generated as output in the example is exactly what I want for my frontend JSON.
Because of the recursion I don't get, how to store it in an array outside of the function.
My goal is a final return as serzializeJson(array).

How to get ordered results from couchbase

I have in my bucket a document containing a list of ID (childList).
I would like to query over this list and keep the result ordered like in my JSON. My query is like (using java SDK) :
String query = new StringBuilder().append("SELECT B.name, META(B).id as id ")
.append("FROM" + bucket.name() + "A ")
.append("USE KEYS $id ")
.append("JOIN" + bucket.name() + "B ON KEYS ARRAY i FOR i IN A.childList end;").toString();
This query will return rows that I will transform into my domain object and create a list like this :
n1qlQueryResult.allRows().forEach(n1qlQueryRow -> (add to return list ) ...);
The problem is the output order is important.
Any ideas?
Thank you.
here is a rough idea of a solution without N1QL, provided you always start from a single A document:
List<JsonDocument> listOfBs = bucket
.async()
.get(idOfA)
.flatMap(doc -> Observable.from(doc.content().getArray("childList")))
.concatMapEager(id -> bucket.async().get(id))
.toList()
.toBlocking().first();
You might want another map before the toList to extract the name and id, or to perform your domain object transformation even maybe...
The steps are:
use the async API
get the A document
extract the list of children and stream these ids
asynchronously fetch each child document and stream them but keeping them in original order
collect all into a List<JsonDocument>
block until the list is ready and return that List.

Sharepoint 2010 Blog - Order rest query by Category

I've created a blog on Sharepoint 2010 and want to query the list via REST for reporting. I want to order the list by the default field Category (internal name PostCategory). Unfortunately, this is a multiselect field, therefore a simple "?$orderby=Category" doesn't work. I've also tried to expand the Category, but that doesn't work either.
Is there a chance, that I can order the list using rest? What about more then one selected Category? Can it be ordered by the first category, then the second, etc.?
If it's not possible using REST, what about ordering within JSON? I use a small javascript, that puts the list in a reporting format. Can I order within the JSON result?
Here is an example:
// Create REST-API URL
var strURL = "<REST-URL>";
// Get information from REST-API and create html output
$.getJSON(strURL, function(data) {
<Create output>
};
// Append to webpart
$('#<WebPartTitle>').append($(html));
EDIT: I've posted the question also here, since it's happening all in sharepoint
Category field (PostCategory internal name) is a multiple choice field, in SharePoint REST it is not supported to apply $orderby query option to this type of field.
But you could sort returned items using JavaScript.
The following example demonstrates how to order Posts by Category field.
There is one important note here:
Since Category field is a multiple choice field value, it is
assumed that only one category could be specified per post.
For that purpose FirstCategoryTitle property is introduced which
represent the title of first category in post item. This property is used > for sorting items
Example
var endpointUrl = 'http://contoso.intranet.com/blog/_vti_bin/listdata.svc/Posts?$expand=Category';
$.getJSON(endpointUrl, function(data) {
var items = data.d.results.map(function(item){
item.FirstCategoryTitle = (item.Category.results.length > 0 ? item.Category.results[0].Title : ''); //get first category
return item;
});
items.sort(postComparer); //sort by category
items.forEach(function(item){
console.log(item.Title);
});
});
function postComparer(x,y) {
return x.FirstCategoryTitle > y.FirstCategoryTitle;
}

Use JSON Input step to process uneven data

I'm trying to process the following with an JSON Input step:
{"address":[
{"AddressId":"1_1","Street":"A Street"},
{"AddressId":"1_101","Street":"Another Street"},
{"AddressId":"1_102","Street":"One more street", "Locality":"Buenos Aires"},
{"AddressId":"1_102","Locality":"New York"}
]}
However this seems not to be possible:
Json Input.0 - ERROR (version 4.2.1-stable, build 15952 from 2011-10-25 15.27.10 by buildguy) :
The data structure is not the same inside the resource!
We found 1 values for json path [$..Locality], which is different that the number retourned for path [$..Street] (3509 values).
We MUST have the same number of values for all paths.
The step provides Ignore Missing Path flag but it only works if all the rows misses the same path. In that case that step acts as as expected an fills the missing values with null.
This limits the power of this step to read uneven data, which was really one of my priorities.
My step Fields are defined as follows:
Am I missing something? Is this the correct behavior?
What I have done is use JSON Input using $.address[*] to read to a jsonRow field the full map of each element p.e:
{"address":[
{"AddressId":"1_1","Street":"A Street"},
{"AddressId":"1_101","Street":"Another Street"},
{"AddressId":"1_102","Street":"One more street", "Locality":"Buenos Aires"},
{"AddressId":"1_102","Locality":"New York"}
]}
This results in 4 jsonRows one for each element, p.e. jsonRow = {"AddressId":"1_101","Street":"Another Street"}. Then using a Javascript step I map my values using this:
var AddressId = getFromMap('AddressId', jsonRow);
var Street = getFromMap('Street', jsonRow);
var Locality = getFromMap('Locality', jsonRow);
In a second script tab I inserted minified JSON parse code from https://github.com/douglascrockford/JSON-js and the getFromMap function:
function getFromMap(key,jsonRow){
try{
var map = JSON.parse(jsonRow);
}
catch(e){
var message = "Unparsable JSON: "+jsonRow+" Desc: "+e.message;
var nr_errors = 1;
var field = "jsonRow";
var errcode = "JSON_PARSE";
_step_.putError(getInputRowMeta(), row, nr_errors, message, field, errcode);
trans_Status = SKIP_TRANSFORMATION;
return null;
}
if(map[key] == undefined){
return null;
}
trans_Status = CONTINUE_TRANSFORMATION;
return map[key]
}
You can solve this by changing the JSONPath and splitting up the steps in two JSON input steps. The following website explains a lot about JSONPath: http://goessner.net/articles/JsonPath/
$..AddressId
Does in fact return all the AddressId's in the address array, BUT since Pentaho is using grid rows for input and output [4 rows x 3 columns], it can't handle a missing value aka null value when you want as results return all the Streets (3 rows) and return all the Locality (2 rows), simply because there are no null values in the array itself as in you can't drive out of your garage with 3 wheels on your car instead of the usual 4.
I guess your script returns null (where X is zero) values like:
A S X
A S X
A S L
A X L
The scripting step can be avoided same by changing the Fields path of the first JSONinput step into:
$.address[*]
This is to retrieve all the 4 address lines. Create a next JSONinput step based on the new source field which contains the address line(s) to retrieve the address details per line:
$.AddressId
$.Street
$.Locality
This yields the null values on the four address lines when a address details is not available in an address line.