NOTE: We've recently changed the way new postings are presented on this site. The "What's new and Hot" section has been transformed to "In the Spotlight" BLOG entries. Even though the ideas in this article are still "valid", if you want to grab the *Current Headlines* from the About Delphi Programming site, please visit the "About Delphi Programming *Current Headlines* sticker" page!

There's one section of this site that brings you the latest news: compilation of hot, new and updated materials on the About Delphi Programming site. This page is located at "What's New and Hot". Every time you visit the site I strongly suggest you to open that page and see if some new items were added to the site.

To enable Delphi programming related site web masters to add this valuable information to their pages, I've developed the "About Delphi Programming *NEW and HOT* sticker". It's a java script file, web masters need to include in their web pages to give their viewers the ultimate source of professional Delphi / Kylix programming content, updated frequently.
 

This java script file (.js) is created by a Delphi developed utility. This application downloads the What's New and Hot page from this site, then uses some regular expressions to extract the correct data from the page and finally creates a .js file using the extracted data.

The idea of this article is to show you the techniques used to download a page from the Internet, do some page scraping and finally present the information in more "situation-friendly" manner.

The key to data extraction methods described in this article is to convert the existing HTML document to more "situation-friendly" source. These are the steps we'll be discussing:

• Retrieval of HTML source documents
• Processing the HTML document, removing the unneeded data
• Transforming the result to string type variables
• Displaying the information extracted in a ListView

Note: The Sticker java script file described above uses the same techniques we are to discuss in this article - it is just enriched with HTML tags, due to the fact that JavaScript document.write method adds well formatted HTML code to an existing document. What's important is that the core of the Sticker .js is the data I'll show you how to extract.

Preparing the Delphi Project
To keep up with the article, I suggest you to start Delphi, create a new project with one blank form. On this form place one TButton (Standard palette) and one TListView (Win32 palette) component. Leave the default component names as Delphi suggests. That is, Button1 for the button and ListView1 for the list view component. You'll use Button1 to get the file from the Internet, do information retrieval and show the result in the ListView1. Also, make sure to add 4 columns to the ListView1: Title, URL, Description, When/Where. The ViewStyle of the ListView1 should be set to vsReport.

   Retrieval of HTML source documents
Before we start extracting data from an HTML file, we need to make sure we have one locally.
Your first task is to create a Delphi function used to download a file from the Internet. One way of achieving this task is to use the WinInet API calls. Delphi gives us full access to the WinInet API (wininet.pas) which we can use to connect to and retrieve files from any Web site that uses either Hypertext Transfer Protocol (HTTP) or File Transfer Protocol (FTP). I've already written an article that describes this technique: Get File From the Net.

Another approach, if you have Delphi 6, is to use the TDownloadURL object. The TDownloadURL object, defined in ExtActns.pas unit, is designed for saving the contents of a specified URL to a file. Here's the code that uses the TDownloadURL to download the "What's New and Hot" page from this site.

function Download_HTM(const sURL, sLocalFileName:string): boolean;
begin
  Result:=True;
  with TDownLoadURL.Create(nil) do
  try
    URL:=sURL;
    Filename:=sLocalFileName;
    try
      ExecuteTarget(nil);
    except
      Result:=False
    end;
  finally
    Free;
  end;
end;

This function, Download_HTM, downloads a file from the URL specified in the sURL parameter, and saves this file locally under a sLocalFileName name. The function returns True if it succeeds, False otherwise. Of course, this function is to be called from the Button1 OnClick event handler. You can see the code below. Note that, locally, the file is saved as c:\temp_adp.newandhot.

procedure TForm1.Button1Click(Sender: TObject);
const
  ADPNEWHOTURL='http://delphi.about.com/cs/newandhot/index.htm';
  TmpFileName='c:\temp_adp.newandhot';
begin
  if NOT Download_HTM(ADPNEWHOTURL,TmpFileName) then
  begin
    ShowMessage('Error in HTML file download');
    Exit;
  end;

  {
  more code to be added
  }

end;

Note: In the process of downloading a file, the TDownloadURL periodically generates an OnDownloadProgress event, so that you can provide users with feedback about the process. I'll leave this for you to implement.

Now, that we have the HTM page locally on the disk we can use the techniques for handling ASCII files from Object Pascal code.

   Processing the HTML document
The next step is to locate the interesting data inside the HTML document and extract it. Since the HTML document is just pure ASCII file, you can use the set of routines designed to work with text files in Delphi.

Before we move on
It's important to have in mind that techniques described in this article are somehow deprecated with "new" intelligent information retrieval techniques like HTML to XML using the XSLT. If you do not know what I'm talking about don't worry.

To be able to successfully extract the data from a web page, the use of some kind of regular expressions for pattern matching is required. This in particular means that you will be able to do page scraping if and only if you know the structure of an HTML document. This is not a big problem if you are the one that creates the web page. Even if you are not the person behind a web page, you can use pattern matching, but must be sure to check your code occasionally, since HTML is dynamic content and a document structure can change very often due to the various banner ad systems and dynamic server-side scripting engines.

In situations when pattern matching is not giving results you can turn to more intelligent solutions like transforming the HTML document to XML - a standard for marking up structured documents; however this is not something we are to discuss here.

If you open up the downloaded file with the Notepad, you should notice that the information we want to extract is placed inside <!--DCnt--> and </!--DCnt--> tags. After you extract that part, you have to make sure any server or client side scripting is excluded - such text usually appears between the <script> and </script> tags. What remains is an HTML code, with 10 items formatted like:

Now, this "item" holds 4 pieces of interesting information. In the code, this item is a held in the ItemBuf string variable. Marked red is the URL of the particular news item. Marked blue is the title of this item. The description is green, and the date and location is maroon.

To get the particular element of information, you can use the following code:

//find the title
iStart:=Pos('<b>',ItemBuf) + Length('<b>');
iStop:=Pos('</b>',ItemBuf);
sTitle:= Copy(ItemBuf, iStart, iStop-iStart);

Finally you transform each item to 4 string type variables and display the information in ListView.

I'm not going to bother you with the project details here, be sure to see the entire code, you'll have plenty to play with.

This is the project at run-time:

If you have any questions or comments to this article, please post them on the Delphi Programming Forum.