pull data with HtmlAgilityPack Web Scraping c#

What could be the reason that it returns null in selectnodes?

HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("https://www.wired.com/most-popular/");
var headerNames = doc.DocumentNode.SelectNodes("//a[@class='archive-item-component__title']").ToList();
string listData = "";
foreach(var item in headerNames)
{
   listData += Environment.NewLine + item.InnerText;   
}
Console.WriteLine(listData);

Here is Solutions:

We have many solutions to this problem, But we recommend you to use the first solution because it is tested & true solution that will 100% work for you.

Solution 1

The reason is that ‘archive-item-component__title’ class in the web site is specified for h2 tags, not for a tags.

Selecting your headerNames like this works:

doc.DocumentNode.SelectNodes("//h2[@class='archive-item-component__title']");

PS: you don’t need to call .ToList() on the SelectNodes results, you can loop through the HtmlNodeCollection as is.

Note: Use and implement solution 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply