Geek Files » PHP Code Examples

Writing Website Scrapers in PHP

admin — Tue, 26 Feb 2008 15:35:01 +0000

This article discusses about how to write a website scraper using PHP for web site data extraction. The concepts taught can be applied and programmed in Java, C#, etc. Basically any language that has a powerful string processing capability. This article will teach you the basics of website scraping. The article will further cover a tutorial to find web ranking from Yahoo.com search engine.

Steps involved to write a scraping program

Visit the URL
Understand the pattern
Validate the structure of pattern on different URLs
Write the program
Test the program using various input parameters

Lets visit each of these steps one bit at a time.

1. Visit the URL

For this tutorial, we will extract Yahoo’s “Today’s Top Searches” section towards the end of their home page (http://www.yahoo.com/).

2. Understand the pattern

Before you begin to write a web scraping program, its important to understand the pattern of the data that you wish to extract. View the page source to understand the pattern. Mentioned below.

The string of text that we should parse is given below:

 

Heidi Klum
Sarah Larson
Oscar Videos
Brad Renfro
Gary Busey
Barack Obama
Razzie Awards
Raisin in the Sun
Stay Home Moms
Net Neutrality

The pattern is that each Search Phrase is enclosed within a

tag. Therefore, we should parse everything between of this text piece to get the text.

3. Validate the structure of pattern on different URLs

If you are writing a script to fetch data that has pagination, you should remember to validate the structure on 3 – 4 pages before you start developing code. The reason behind doing this is that the presentation of the first page could differ in subsequent pages.

4. Write the program

You could use any programming language like Java, C#, PHP, PERL, etc. for this processing. I have used PHP for this example.

 
//change fso() to f sock open (my blog was causing an error)
//change fwt() to f write (my blog was causing an error)
//change fgs() to f gets (my blog was causing an error)
//change fc() to f close (my blog was causing an error)
 
$fp = fso("www.yahoo.com", 80, $errno, $errstr, 30);
if (!$fp) {
    echo "$errstr ($errno)
\n";
} else {
	$out = "GET / HTTP/1.1\r\n";
	$out .= "Host: www.yahoo.com\r\n";
	$out .= "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12\r\n";
	$out .= "Connection: Close\r\n\r\n";
 
	$str = "";
 
	fwt($fp, $out);
	while (!feof($fp)) {
		$str .= fgs($fp, 1024);
	}
	fc($fp);
}
 
$pos = strpos($str, "\"popsearchbd\"");
$pos = $pos + strlen("\"popsearchbd\"");
 
if($pos == false) {
	echo "No information available";
}
else {
 
	while(1) {
		$pos = strpos($str, "fp-buzzmod\">", $pos);
 
		if($pos === false) {
			break;
		}
 
		$pos = $pos + strlen("fp-buzzmod\">");
		$temppos = $pos;
		$pos = strpos($str, "", $pos);
 
		$datalength = $pos - $temppos;
 
		$data = substr($str, $temppos , $datalength);
		echo $data;
		echo "\n";
	}
 
}

5. Test the program using various input parameters

You should test your program for all the parameters that the web page can take. I have experienced change in layout & data based on the parameters that are passed.

Notes on processing forms and cookies

Some pages use form data and cookies to render data. In such cases you should remember to check the Request and Response headers and identify what is necessary to get the results that you want. If the page requires a cookie value, you should then use the cookie information in your Request Headers. Look at the note below that I use to inspect Request and Response headers.

Tool to inspect Request and Response Headers

I use Live HTTP Headers (a plug in for FireFox) to check for Request and Response Headers. Visit http://livehttpheaders.mozdev.org for more details. To install this plug in visit http://livehttpheaders.mozdev.org/installation.html and click on the ‘download it’ link on the latest release. Please read the release notes before installing a particular version.

Please feel free to use the comments section down to share other tools that you guys use to monitor Request and Response Headers.

Future Maintenance of the program

From a maintenance perspective you should monitor the page frequently and re-validate the HTML structure. The reason for this is because nothing in this world is constant and so is the website. Design changes could result in a change in the HTML code. I recommend scheduling this activity for at least once a month.

This is all that is there in this tutorial. In the next tutorial, I will guide you step-by-step on how to create a program to check for yahoo.com search engine ranking. Use the subscribe form below to keep yourself updated when the tutorial goes live.

Would you like to check out some of my PHP5 Tutorials as well?

PHP 5 Tutorial – Handling Exceptions in PHP5

admin — Thu, 14 Feb 2008 08:58:21 +0000

In this tutorial we will cover the following:

What is an exception?
The use of a try…catch block
Anatomy of PHP5 Exception class
Extending the Exception class
A note on unhanded exceptions

What is an exception?

An exception is a logical/system error that occurs during the normal execution of a script. The exception could either be raised by the system or the program itself it the exception cannot be handled and the caller script/function needs to be informed about the same.

The use of a try…catch block

PHP5 introduces the try…catch block to trap exceptions. Look at the example below.

 
try {
   check();
}
catch(Exception $e) {
   echo "Message : " . $e->getMessage();
   echo "Code : " . $e->getCode();
}
 
function check() {
   if(some error condition) {
      throw new Exception("Error String",Error Code);
   }
}

In the above example, the method check() is called between the try {} block. The try{} block is the area where you will place your code that could raise an exception. Below the try{} block is the catch() {} block. The catch block expects the Exception type of object as a parameter. Within the catch() {} block you will place your logic to either fix the issue or log the error.

In the function check(), we raise an Exception using the ‘throw’ keyword. The statement following ‘throw’ is the syntax of creating a new object of Exception type. The exception class accepts two parameters. The left parameter is a string that is the error message and the right parameter is the integer error code that you wish to assign to that error.

Anatomy of PHP5 Exception class

 
class Exception {
	protected $message;
	protected $code;
	protected $file;
	protected $line;
 
	private $string;
	private $trace;
 
	public function __construct($message = null, $code = 0);
	public function __toString();
 
	final public function getCode();
	final public function getMessage();
	final public function getFile();
	final public function getLine();
	final public function getTrace();
	final public function getTraceAsString();
	final private __clone();
}

In the above example, except for __construct and __toString(), no other method can be overridden as all other methods are ‘final’.

Related Reading:

PHP5 Tutorial – Creating a PHP5 Constructor __construct()

PHP5 Tutorial – Defining Attributes of a PHP5 Class

PHP5 Tutorial OOPS – PHP5 Class Access Specifiers – public, private and protected

PHP 5 Tutorial – Final Class and Methods

PHP5 Tutorial – Magic Methods – __toString() method

Extending the Exception class

You can also extend the exception class as follows:

 
class CustomerException extends Exception {
 
  public function __construct($message = null, $code = 0) {
 
    $t_message = "Exception raised in CustomerException ";
    $t_message .= "with message : " . $message;
 
    parent::__construct($t_message, $code);
 
    }
 
}
 
 
function testException() {
 
	throw new CustomerException("CustomerException has been raised",101);
 
}
 
try {
	testException();
}
catch(CustomerException $e) {
	echo "Error Message : " $e->getMessage();
	echo "Error Code : " $e->getCode();
}

Output:

Error Message : CustomerException has been raised
Error Code : 101

Related Reading:
PHP5 Tutorial – Inheritance

A note on unhanded exceptions

All un-handled exceptions are passed up the function stack order till the time either a try…catch block is not made available. If a try…catch block is unavailable it is passed to the PHP core and the program execution stops there.

Please feel free to write back for any clarification.

Subscribe to my free newsletter:

Related Posts on PHP5 Tutorial – Object Oriented Programming (OOPS)

PHP Script to print nested array

admin — Fri, 16 Nov 2007 05:28:35 +0000

Here is a script that will allow you to print a nested array upto any level. Can I request you to please review this and let me know how this can be further optimized.

$array["sunil"] = "Bhatia";
$array[0] = array("one"=>"two",array(1,2,3,array(4,5,6)));
$array["sunil2"] = "Bhatia2";
 
processArray($array);
 
$count=0;
 
function processArray($array) {
 
	global $count;
 
	$count++; //this variable is for calculating tab space
 
	if(is_array($array) === true) {
 
		foreach($array as $key => $value) {
			if(is_array($array[$key]) === true) {
				processArray($array[$key]);
			}
			else {
				for($i = 1 ; $i < $count ; $i++) echo "  "; //change this to space ' ' if running from CLI
				echo $value."
";
			}
		}
	}
 
	$count--;
}

Related Posts on PHP5 Tutorial – Object Oriented Programming (OOPS)

PHP Script to Extract Email Address from any text

admin — Wed, 14 Nov 2007 12:14:50 +0000

I have developed a function that you can embed in your PHP applications that will help you extract email addresses from a given piece of text.

I have tested this on a string of (actually 4 – 5 paragraphs) text and this has performed very well.

Please feel free to use this code in your applications and let me know if you face any issues.

 
$text = ' email@domain.com ';
 
function parseTextForEmail($text) {
	$email = array();
	$invalid_email = array();
 
	$text = ereg_replace("[^A-Za-z._0-9@ ]"," ",$text);
 
	$token = trim(strtok($text, " "));
 
	while($token !== "") {
 
		if(strpos($token, "@") !== false) {
 
			$token = ereg_replace("[^A-Za-z._0-9@]","", $token);
 
			//checking to see if this is a valid email address
			if(is_valid_email($email) !== true) {
				$email[] = strtolower($token);
			}
			else {
				$invalid_email[] = strtolower($token);
			}
		}
 
		$token = trim(strtok(" "));
	}
 
	$email = array_unique($email);
	$invalid_email = array_unique($invalid_email);
 
	return array("valid_email"=>$email, "invalid_email" => $invalid_email);
 
}
 
function is_valid_email($email) {
	if (eregi("^[_a-z0-9-]+(\.[_a-z0-9-]+)*@[a-z0-9-]+(\.[a-z0-9-]+)*(\.([a-z]){2,4})$",$email)) return true;
	else return false;
}
 
var_dump(parseTextForEmail($text));

In the above program, you should pass a string of text to the method parseTextForEmail() that will return an array with valid and invalid email addresses.

Check the code and let me know if you have any issues or suggestions for improvements.

You can also check my PHP Tutorials – Starting with Creating a PHP5 Class