Feature Article:

The Secret Benefit Of Accessibility Part 1: Increased Usability
Web accessibility has so many benefits that I really do wonder why such a large number of websites have such diabolically bad accessibility. One of the main benefits is increased usability, which according to usability guru, Jakob Nielson, can...
...Read More

Screen Scraping Your Way Into RSS

Additional Reading


RSS is one the hottest technologies at the moment, and even big web publishers (such as the New York Times) are getting into RSS as well. However, there are still a lot of websites that do not have RSS feeds.

If you still want to be able to check those websites in your favourite aggregator, you need to create your own RSS feed for those websites. This can be done automatically with PHP, using a method called screen scrapping. Screen scrapping is usually frowned upon, as it's mostly used to steal content from other websites.

I personally believe that in this case, to automatically generate a RSS feed, screen scrapping is not a bad thing. Now, on to the code!

Getting the content

For this article, we'll use PHPit as an example, despite the fact that PHPit already has RSS feeds (http://www.phpit.net/syndication/).

We'll want to generate a RSS feed from the content listed on the frontpage (http://www.phpit.net). The first step in screen scraping is getting the complete page. In PHP this can be done very easily, by using implode(file("", "[the url here]")); IF your web host allows it. If you can't use file() you'll have to use a different method of getting the page, e.g. using the CURL library (http://www.php.net/curl).

Now that we have the content available, we can parse it for the content using some regular expressions. The key to screen scraping is looking for patterns that match the content, e.g. are all the content items wrapped in <div>'s or something else? If you can successfully discover a pattern, then you can use preg_match_all() to get all the content items.

For PHPit, the pattern that match the content is <div class="contentitem">[Content Here]<div>. You can verify this yourself by going to the main page of PHPit, and viewing the source.

Now that we have a match we can get all the content items. The next step is to retrieve the individual information, i.e. url, title, author, text. This can be done by using some more regular expression and str_replace() on the each content items.

By now we have the following code;


// Get page

$url = "http://www.phpit.net/";

$data = implode("", file($url));

// Get content items

preg_match_all ("/<div class="contentitem">([^`]*?)</div>/", $data, $matches);

Like I said, the next step is to retrieve the individual information, but first let's make a beginning on our feed, by setting the appropriate header (text/xml) and printing the channel information, etc.


// Begin feed

header ("Content-Type: text/xml; charset=ISO-8859-1");

echo "<?xml version="1.0" encoding="ISO-8859-1" ?>


<rss version="2.0"






<title>PHPit Latest Content</title>

<description>The latest content from PHPit (http://www.phpit.net), screen scraped!</description>




Now it's time to loop through the items, and print their RSS XML. We first loop through each item, and get all the information we get, by using more regular expressions and preg_match(). After that the RSS for the item is printed.


// Loop through each content item

foreach ($matches[0] as $match) {

// First, get title

preg_match ("/">([^`]*?)</a></h3>/", $match, $temp);

$title = $temp['1'];

$title = strip_tags($title);

$title = trim($title);

// Second, get url

preg_match ("/<a href="([^`]*?)">/", $match, $temp);

$url = $temp['1'];

$url = trim($url);

// Third, get text

preg_match ("/<p>([^`]*?)<span class="byline">/", $match, $temp);

$text = $temp['1'];

$text = trim($text);

// Fourth, and finally, get author

preg_match ("/<span class="byline">By ([^`]*?)</span>/", $match, $temp);

$author = $temp['1'];

$author = trim($author);

// Echo RSS XML

echo "<item>

echo " <title>" . strip_tags($title) . "</title>

echo " <link>http://www.phpit.net" . strip_tags($url) . "</link>

echo " <description>" . strip_tags($text) . "</description>

echo " <content:encoded><![CDATA[

echo $text . "

echo " ]]></content:encoded>

echo " <dc:creator>" . strip_tags($author) . "</dc:creator>

echo " </item>



And finally, the RSS file is closed off.



That's all. If you put all the code together, like in the demo script, then you'll have a perfect RSS feed.


In this tutorial I have shown you how to create a RSS feed from a website that does not have a RSS feed themselves yet. Though the regular expression is different for each website, the principle is exactly the same.

One thing I should mention is that you shouldn't immediately screen scrape a website's content. E-mail them first about a RSS feed. Who knows, they might set one up themselves, and that would be even better.

Download sample script at http://www.phpit.net/viewsource.php?url=/demo/screenscrape%20rss/example.php

Dennis Pallett is a young tech writer, with much experience in ASP, PHP and other web technologies. He enjoys writing, and has written several articles and tutorials. To find more of his work, look at his websites at http://www.phpit.net, http://www.aspit.net and http://www.ezfaqs.com


More Reading:

Superior to Dreamweaver

TEN Tips For Your Web Site Home Page

Open Source Free Software De Mystified

50 Surefire Web Design Tips

7 Steps to Web Sites that Sell

Get It All with Good Web Content

Web Site Monitoring Is A Global Necessity

Metrics Matter

The Reality of Successful Web Site Today

Are your SEO efforts going to waste

Superior to Dreamweaver?
I've heard a lot of good comments regarding Macromedia's Dreamweaver site building tool. Does anyone use Site Build It!?, It's a website building tool that includes hosting and allows you to research profitable keywords, run a search engine...
...Read More

TEN Tips For Your Web Site Home Page
1. Loading time: try to be below 20kb less number of images specify height and width of images Html with out erros WYSIWYG editors load up your html code. so try using text editor part of your html editors instead of visual editor ...
...Read More

Open Source Free Software De-Mystified
Did you know that the most powerful software and web applications are all free? Yes that's right, just about every commercial software or web application you know of is most likely available as a free Open Source project/clone. "Why haven't...
...Read More

50 Surefire Web Design Tips
Tips to brand your website Include your logo in all pages. Position it at the top left or each page. Complement your logo with a tagline or catchy sentence that summarizes your business purpose. For example "Always low prices" is...
...Read More

7 Steps to Web Sites that Sell
Your web site is like a fight of stairs into your business. Once you've got prospects to your home page - your online front door - you want to move them to action. If you miss a step or two, prospects will fall and won't make it in the door to...
...Read More