Skip to Content
Home » Questions » problems webscraping with RCurl

problems webscraping with RCurl

Submitted by Sven Duve on May 28, 2012

I basically have two questions/ problems any help is highly appreciated. I am trying to webscrape the following page:

http://toast.gasunie.de/Pages/search.aspx?lang=en&soid=GUD

I tried the following:

webpage = getURL(url)
page = postForm(url, GasQuality="L",
                   .opts = curlOptions(
                   referer="http://toast.gasunie.de/gud/search.aspx?soid=GUD&lang=de",
                   verbose = TRUE,
                   header = TRUE,
                   followLocation = TRUE,
                   useragent = "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13"
              ))
 
cat(page, file = "page.html") 
shell.exec("page.html")

If you look at the HTML source, you can see that I omitted some of the name=value pairs. but including them doesnt make a difference either.