Data mining and content aggregation with PHP

Tag it:

This is an interesting article I came across. It is mirrored and the link is provided at the end of the page.

The functions described in this article will allow you to parse unstructured data from HTML pages using PHP and perl regular expressions,

The unstructured data can then be stored in a structured fashion in a database of your choosing.

Data Mining Basics Tutorial:

Luckily, there are alternatives. Using PHP and MySQL you can effectively accomplish the same content aggregation tasks with little cost, but you have to learn the basics first. Most data mining projects follow these basic steps.

Basic Data Mining Steps

Fetch the HMTL page(s) of Interest using the Snoopy PHP Class
Split the page HTML into a more manageable portion
Remove un-wanted HTML tag attributes
Reformat HTML, adjust spacing and remove entities
Match content with regular expressions
Store content into a MySQL database for future use

Step 1. - Fetching the Data / HTML Content

Developers have done this work already. The Snoopy PHP class, which can be downloaded at http://sourceforge.net/projects/snoopy/, has all the necessary tools to download an HTML page from the internet. It's advisable to be considerate when you are retrieving content from any site. Contact the Web site admin before fetching with any automated scripting, don't bombard the Web site with thousands of HTTP requests a second, always take notice of any copyrights and setup a contract for content sharing if needed. Leeching Web content from a Web site without permission can lead to serious legal issues.

Steps 2, 3 & 4 - Parsing the Data

Once you have the HTML page(s), you will need to parse the data into a friendlier format. This is done so that pattern matching will be more reliable and consistent in the future. The functions shown below will greatly help in this matter.

PHP Data Parsing Functions:

<div><p><br />' . '</p><h1><b><i><a></a></i></b><ul><li><pre><hr><blockquote><b><i><a><img>';
        $source = strip_tags($source, $allowedTags);
        return preg_replace('/<(.*?)>/ie', "'<'.removeAttributes('\\1').'>'", $source);
    }
     
     
    //
    // Remove unwanted attributes from HTML source using the PHP preg_replace function
    //
    function removeAttributes($source) {
        global $stripAttrib;
        // Disallow these attributes/prefix within a tag
        if (empty($stripAttrib)) $stripAttrib = 'javascript:[\S]+|onclick[\s\t\r\n]*=[\s\t\r\n]*[\S]+|ondblclick|onmousedown[\s\t\r\n]*=[\s\t\r\n]*[\S]+|onmouseup[\s\t\r\n]*=[\s\t\r\n]*[\S]+|onmouseover[\s\t\r\n]*=[\s\t\r\n]*[\S]+|' . 'onmousemove[\s\t\r\n]*=[\s\t\r\n]*[\S]+|onmouseout[\s\t\r\n]*=[\s\t\r\n]*[\S]+|onkeypress[\s\t\r\n]*=[\s\t\r\n]*[\S]+|onkeydown[\s\t\r\n]*=[\s\t\r\n]*[\S]+|onkeyup[\s\t\r\n]*=[\s\t\r\n]*[\S]+|' . 'valign[\s\t\r\n]*=[\s\t\r\n]*[\S]+|align[\s\t\r\n]*=[\s\t\r\n]*[\S]+|alt[\s\t\r\n]*=[\s\t\r\n]*[\S]+|border[\s\t\r\n]*=[\s\t\r\n]*[\S]+|' . 'cellpadding[\s\t\r\n]*=[\s\t\r\n]*[\S]+|cellspacing[\s\t\r\n]*=[\s\t\r\n]*[\S]+|width[\s\t\r\n]*=[\s\t\r\n]*[\S]+|height[\s\t\r\n]*=[\s\t\r\n]*[\S]+|background[\s\t\r\n]*=[\s\t\r\n]*[\S]+|src[\s\t\r\n]*=[\s\t\r\n]*[\S]+|style[\s\t\r\n]*=[\s\t\r\n]*[\S]+';
        return stripslashes(preg_replace("/[\s\t\r\n]+$stripAttrib/i", '', $source));
    }
     
     
    function reformatHTML($document) {
        // $document should contain an HTML document.
        // This will remove HTML tags, javascript sections
        // and white space. It will also convert some
        // common HTML entities to their text equivalent.
         
        $search = array ("'<script[^>]*?>.*?'si", // Strip out javascript
        "'([\r\n]{1,})[\t\s]+'", // Strip out white space
        "'&(quot|#34);'i", // Replace HTML entities
        "'&(amp|#38);'i",
            "'&(lt|#60);'i",
            "'&(gt|#62);'i",
            "'&(nbsp|#160);'i",
            "'&(iexcl|#161);'i",
            "'&(cent|#162);'i",
            "'&(pound|#163);'i",
            "'&(copy|#169);'i",
            "'[\s\t]+'",
            "'[\s]+>'", // Strip Spaces at End of Tags
        "'<tr'i", split="" table="" rows="">'i", // Split Table Rows
        "'<br />'i", // Add Spacing
        //"'&#(\d+);'e",     // evaluate as php
        );
        $replace = array ("",
            "\\1",
            "\"",
            "&",
            "<",
            ">",
            " ",
            chr(161),
            chr(162),
            chr(163),
            chr(169),
            " ",
            ">",
            "\n\n<tr",>\n\n",
            "\n<br />",
            //"chr(\\1)",
        );
        return preg_replace($search, $replace, $document);
    }
     
    //
    // Split the page HTML
    //
    function splitPageHTML($document, $start_pattern, $end_pattern) {
         
        if (!empty($start_pattern)) {
            $start_split = preg_split($start_pattern, $document, 2);
            if (count($start_split) > 1) {
                //var_dump($start_split);
                $document = "";
                for ($i = 1; $i <= count($start_split)-1; $i++) $document .= $start_split[$i];
            }
        }
        if (!empty($end_pattern)) {
            $end_split = preg_split($end_pattern, $document, 2);
            if (count($end_split) > 1) {
                //var_dump($end_split);
                $document = "";
                for ($i = 0; $i <= count($end_split)-2; $i++) $document .= $end_split[$i];
            }
        }
        return $document;
    }
     
     
     
    /////////////////////////////////////////////////
     
     
     
    function strip_leading_and_trailing_double_quotes($string) {
        while (substr($string, -1) == '"' || substr($string, 0, 1) == '"') {
            if (substr($string, -1) == '"') {
                $string = substr($string, 0, strlen($string)-1);
            } // remove " from end of string
            if (substr($string, 0, 1) == '"') {
                $string = substr($string, 1, strlen($string)-1);
            } // remove " from start of string
        }
        return $string;
    }
     
     
    function format_text_field($value, $width) {
        if (!$width) {
            $width = 30;
        }
        $new_width = $width - strlen($value);
        $new_value = $value;
        for ($i = 0; i < $width; $i++) {
            $new_value .= " ";
        }
        return ($new_value);
    }
     
     
    function truncate_string($string, $length = 70)
    // This function will truncate a string and add "..." to a specified length.
    {
        if (strlen($string) > $length) {
            // Reduce to $length-3 characters and add "..."
            $split = preg_split("/\n/", wordwrap($string, $length-3));
            $string = $split[0] . "...";
        }
        return ($string);
    }
     
     
    function ucfirst_title($string) {
        // Split the words (\W) by delimiters, ucfirst and then recombine with delimiters.
        $temp = preg_split('/(\W)/', $string, -1, PREG_SPLIT_DELIM_CAPTURE );
        foreach ($temp as $key => $word) {
            $temp[$key] = ucfirst(strtolower($word));
        }
        $new_string = join ('', $temp);
        // Do the Search and Replacements on the $new_string.
        $search = array (' And ', ' Or ', ' But ', ' At ', ' In ', ' On ', ' To ', ' From ', ' Is ', ' A ', ' An ', ' Am ', ' For ', ' Of ', ' The ', "'S", ' Cpa', ' Llc', ' Llp', ' Md', ' Dds', ' Psy D', ' Dmd', ' Phd');
        $replace = array (' and ', ' or ', ' but ', ' at ', ' in ', ' on ', ' to ', ' from ', ' is ', ' a ', ' an ', ' am ', ' for ', ' of ', ' the ', "'s", ' CPA', ' LLC', ' LLP', ' MD', ' DDS', ' PSY D', ' DMD', ' PHD');
        $new_string = str_replace($search, $replace, $new_string);
        // Several Special Replacements ('s, McPherson, McBain, etc.) on the $new_string.
        $new_string = preg_replace("/Mc([a-z]{3,})/e", "\"Mc\".ucfirst(\"$1\")", $new_string);
        // Another Strange Replacement (example: "Pure-Breed Dogs: the Breeds and Standards") on the $new_string.
        $new_string = preg_replace("/([:;])\s+([a-zA-Z]+)/e", "\"$1\".\" \".ucfirst(\"$2\")", $new_string);
        // If this is a very low string ( > 60 char) then do some more replacements.
        if (strlen($new_string) > 60) {
            $search = array (" With ", " That ");
            $replace = array (" with ", " that ");
            $new_string = str_replace($search, $replace, $new_string);
        }
        return ($new_string);
    }
     
?>

First, use the splitPageHTML() function to split the page. This can be done by examining the HTML code for comment tags, or just split out the <body> section of the HTML.

Next, use the removeTags() function to remove most of the unwanted HTML tags. You still want to keep the table tags and other structural tags, as they will usually help you in identifying the content.

Finally, use the reformatHTML() function to reformat the HTML and adjust spacing. This will greatly clean up the remaining data and prepare it for pattern matching.

Step 5 - Matching Content with PERL Regular Expressions

This is where it gets more complicated. If you are not familiar with regular expressions, please take a moment to visit one of the following sites.

PHP Builder Tutorial on Regular Expressions
Google Search for Regular Expressions
Or purchase the very informative O'Reilly book Mastering Regular Expressions

Now that you've mastered regular expressions, it's time to look at your content up to this step. Use the PHP vardump() function to output your resulting content to the screen. Take notice of structure HTML tags (table, tr, td, div, p). Also, note any class attributes. Class attributes will greatly help you to target certain patterns. It's beyond the scope of this article to discuss all possibilities, but we can go over a simple example pattern.

Shown below is example PHP code of how to match a phone number from a sniblet of HTML ($document). It's probably one of the simplest regular expression matchings. I'll go over it piece by piece. Note that I use $document as the HTML document text. $phone_pattern is used as the pattern in the PERL regular expression match, PHP preg_match() function.

// HTML sniblet from the var_dump($document) call
// <td class="details"> Phone: (222) 222-2222 </td>

$phone_pattern = '/[\s\n\r\t]*Phone:[\s\n\r\t]*([ 0-9\-]+)[\s\n\r\t]*/i';
if (preg_match($phone_pattern, $document, $matches)) $info['phone'] = $matches[1];

Note that I've shown the class attribute, class="details", as it can sometimes help in the pattern matching scheme. Table structures and other HTML tags may also help. But, in this simple case we are trying to match a phone number. Below are the pieces to this regular expression.

[\s\n\r\t]* - white space, soft return, hard return, tab; 0 or more occurrences
[ 0-9\-]+ - spaces, parenthesis, numbers 0 thru 9, dash; 1 or more occurrences
( ) - matching portions set to $matches[1], $matches[2] and so on

This was just a simple example, but it shows a very important aspect of regular expressions. By using regular expressions you can continue to match your data in the future even if the page changes slightly. In combination with the three functions mentioned above; splitPageHTML(), removeTags() and reformatHTML(), your scripts will continue to match data properly well into the life of the script.

Comments

Only registered users can write comments!

Last Updated ( Friday, 18 July 2008 )