a6e5b969-be6e-4756-ae53-47d3cb899ca4.png

b44614b6-8d07-422f-88ee-1b295f0f28a2.jpg

RPA Developer Lead

26568d2a-f93f-46f9-8bf9-b2027057c866.nupkg

**Problem:**

  

Web scrape HTML pages without having a browser open and without UI Automation.

  

**Solution:**

  

XPath selectors ( [https://www.w3schools.com/xml/xpath\_intro.asp](https://www.w3schools.com/xml/xpath_intro.asp) ) over HTML Agility Pack ([http://html-agility-pack.net/](http://html-agility-pack.net/))

  

**Functional Requirements:**

*   Plain HTML pages (content should not be generated dynamically with JavaScript).

You can get ready-made XPath selectors by using Google Chrome to open the page of interest, right clicking an element, Inspect, right click HTML definition, choose Copy and then "Copy XPath" - then you can adjust that selector, make it more generic, etc.  
  
The **output** is a List of strings.  
  
Test Case 1:  
Attribute: "INNERHTML"  
URL: "[https://www.w3schools.com/xml/xpath\_intro.asp"](https://www.w3schools.com/xml/xpath_intro.asp%22)  
XPath Selector: "//\*\[@id=""main""\]/h2"  
  
Test Case 2:  
Attribute: "href"  
URL: "[https://www.w3schools.com/xml/xpath\_intro.asp"](https://www.w3schools.com/xml/xpath_intro.asp%22)  
XPath Selector: "//\*\[@id=""main""\]/div\[4\]/a\[1\]"

XPath query based web scrape method without using a browser.

Web Scraping with XPath