java - How can I extract a list of id's from a webpage?

Question

I have this code in a web page I am trying to scrape with selenium:

 <h4 class="category" id="sc_14295">...</h4>
 <h4 class="category" id="sc_14292">...</h4>
 <h4 class="category" id="sc_14291">...</h4>
 <h4 class="category" id="sc_14299">...</h4>

I have tried to implement a method like this (which I found here on SO) to get at least one id:

String id = driver.findElement(By.xpath("//div[@class='category']")).getAttribute("id");

in this way:

String id = driver.findElement(By.xpath("//h4[@class='category']")).getAttribute("id");

It doesn't work. I assume I should be using findElements() but that can't have getAtttribute. How can I extract a list of all id's? Thanks.

question from:https://stackoverflow.com/questions/65890247/how-can-i-extract-a-list-of-ids-from-a-webpage

Answer 1

To print the List of id attribute of the elements you need to induce WebDriverWait for the visibilityOfAllElementsLocatedBy() and you can use Java8 stream() and map() and you can use either of the following Locator Strategies:

• cssSelector:

  System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector("h4.category"))).stream().map(element->element.getAttribute("id")).collect(Collectors.toList()));
  

• xpath:

  System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//h4[@class='category']"))).stream().map(element->element.getAttribute("id")).collect(Collectors.toList()));