How to scrap text from internet in VB

Visual Basic Topics
User avatar
Rksk
Major
Major
Posts: 730
Joined: Thu Jan 07, 2010 4:19 pm
Location: Rathnapura, Sri Lanka

How to scrap text from internet in VB

Post by Rksk » Sun Sep 25, 2011 11:11 pm

I have wanted to scrap texts from web pages. I searched a lot keywords but didn't find any working way.
Please help me to get read a file from internet.

Also i found a example to download and save files from internet to local drives.

Code: Select all

Option Explicit
Private Declare Function URLDownloadToFile Lib "urlmon" Alias "URLDownloadToFileA" (ByVal pCaller As Long, ByVal szURL As String, ByVal szFileName As String, ByVal dwReserved As Long, ByVal lpfnCB As Long) As Long
Private Declare Function InternetOpen Lib "wininet" Alias "InternetOpenA" (ByVal sAgent As String, ByVal lAccessType As Long, ByVal sProxyName As String, ByVal sProxyBypass As String, ByVal lFlags As Long) As Long
Private Declare Function InternetCloseHandle Lib "wininet" (ByVal hInet As Long) As Integer


'Purpose     :  Retreview text from a web site
'Inputs      :  sURLFileName            The URL and file name to download.
'               sSaveToFile             The filename to save the file to.
'               [bOverwriteExisting]    If True overwrites the file if it existings
'Outputs     :  Returns True on success.


Function InternetGetFile(sURLFileName As String, sSaveToFile As String, Optional bOverwriteExisting As Boolean = False) As Boolean
    Dim lRet As Long
    Const S_OK As Long = 0, E_OUTOFMEMORY = &H8007000E
    Const INTERNET_OPEN_TYPE_PRECONFIG = 0, INTERNET_FLAG_EXISTING_CONNECT = &H20000000
    Const INTERNET_OPEN_TYPE_DIRECT = 1, INTERNET_OPEN_TYPE_PROXY = 3
    Const INTERNET_FLAG_RELOAD = &H80000000
    
    On Error Resume Next
    'Create an internet connection
    lRet = InternetOpen("", INTERNET_OPEN_TYPE_DIRECT, vbNullString, vbNullString, 0)
    
    If bOverwriteExisting Then
        If Len(Dir$(sSaveToFile)) Then
            VBA.Kill sSaveToFile
        End If
    End If
    'Check file doesn't already exist
    If Len(Dir$(sSaveToFile)) = 0 Then
        'Download file
        lRet = URLDownloadToFile(0&, sURLFileName, sSaveToFile, 0&, 0)
        If Len(Dir$(sSaveToFile)) Then
            'File successfully downloaded
            InternetGetFile = True
        Else
            'Failed to download file
            If lRet = E_OUTOFMEMORY Then
                Debug.Print "The buffer length is invalid or there was insufficient memory to complete the operation."
            Else
                Debug.Assert False
                Debug.Print "Error occurred " & lRet & " (this is probably a proxy server error)."
            End If
            InternetGetFile = False
        End If
    End If
    On Error GoTo 0
    
End Function
But it's a long & slower way to get a text from internet, because we have to save and read a file.
User avatar
Enigma
Lieutenant
Lieutenant
Posts: 74
Joined: Sun Jan 16, 2011 12:40 am
Location: Colombo, Sri Lanka

Re: How to scrap text from internet in VB

Post by Enigma » Tue Sep 27, 2011 10:20 pm

Hi Rksk
I'm not that much a VB guy. But try this function. it will give you a start

Code: Select all

 Function GetWebData(ByVal url As String) As String
        Dim wclient As WebClient
        Dim data As String
        wclient = New WebClient()
        data = wclient.DownloadString(url)
        Return data
    End Function
use the name space "System.Net".

thanks
User avatar
Rksk
Major
Major
Posts: 730
Joined: Thu Jan 07, 2010 4:19 pm
Location: Rathnapura, Sri Lanka

Re: How to scrap text from internet in VB

Post by Rksk » Tue Sep 27, 2011 10:31 pm

Enigma wrote: use the name space "System.Net".
I'm not on .Net bro, still using VB6 :cry:

Anyway thank you for the reply.
User avatar
Enigma
Lieutenant
Lieutenant
Posts: 74
Joined: Sun Jan 16, 2011 12:40 am
Location: Colombo, Sri Lanka

Re: How to scrap text from internet in VB

Post by Enigma » Tue Sep 27, 2011 10:36 pm

ohh I'm sorry RKSK. didn't notice the syntax :cry:
User avatar
Rksk
Major
Major
Posts: 730
Joined: Thu Jan 07, 2010 4:19 pm
Location: Rathnapura, Sri Lanka

Re: How to scrap text from internet in VB

Post by Rksk » Tue Sep 27, 2011 10:42 pm

No problem :P
I was waiting for a reply since Saturday, so your reply was a :yahoo:
User avatar
Enigma
Lieutenant
Lieutenant
Posts: 74
Joined: Sun Jan 16, 2011 12:40 am
Location: Colombo, Sri Lanka

Re: How to scrap text from internet in VB

Post by Enigma » Wed Sep 28, 2011 12:41 am

ok try this

Code: Select all

Option Explicit
Private Declare Function InternetOpen Lib "wininet.dll" Alias "InternetOpenA" (ByVal sAgent As String, ByVal lAccessType As Long, ByVal sProxyName As String, ByVal sProxyBypass As String, ByVal lFlags As Long) As Long
Private Declare Function InternetOpenUrl Lib "wininet.dll" Alias "InternetOpenUrlA" (ByVal hOpen As Long, ByVal sUrl As String, ByVal sHeaders As String, ByVal lLength As Long, ByVal lFlags As Long, ByVal lContext As Long) As Long
Private Declare Function InternetReadFile Lib "wininet.dll" (ByVal hFile As Long, ByVal sBuffer As String, ByVal lNumBytesToRead As Long, lNumberOfBytesRead As Long) As Integer
Private Declare Function InternetCloseHandle Lib "wininet" (ByVal hInet As Long) As Integer

Function InternetGetFile(sURLFileName As String) As String

Const INTERNET_OPEN_TYPE_PRECONFIG = 0
Const INTERNET_FLAG_RELOAD = &H80000000
Dim hinter As Long
Dim hinterurl As Long
Dim data As String * 2038
Dim tempdata As String
Dim bytesrd As Long
Dim result As Integer
Dim IsreadData As Boolean


hinter = 0
hinterurl = 0
hinter = InternetOpen("Mozillz/1.0", INTERNET_OPEN_TYPE_PRECONFIG, vbNullString, vbNullString, 0)
hinterurl = InternetOpenUrl(hinter, sURLFileName, vbNullString, 0, INTERNET_FLAG_RELOAD, 0)

IsreadData = True

While IsreadData
data = vbNullString
result = InternetReadFile(hinterurl, data, Len(data), bytesrd)

If bytesrd = 0 Then
IsreadData = False
End If

tempdata = tempdata + data
Wend
InternetCloseHandle (hinterurl)
InternetGetFile = tempdata
End Function

You need to pass the full URL (eg http://www.google.lk)
User avatar
Rksk
Major
Major
Posts: 730
Joined: Thu Jan 07, 2010 4:19 pm
Location: Rathnapura, Sri Lanka

Re: How to scrap text from internet in VB

Post by Rksk » Wed Sep 28, 2011 12:57 am

:clap: :clap: :clap:
Thank you very much dear.
:yahoo: :yahoo: :yahoo:
User avatar
Neo
Site Admin
Site Admin
Posts: 2642
Joined: Wed Jul 15, 2009 2:07 am
Location: Colombo

Re: How to scrap text from internet in VB6

Post by Neo » Wed Sep 28, 2011 4:26 am

I thought you are submitting an article. Didn't thought it's a question ;)

Okay...Here is one short method. Wininet produces problems behind proxies and firewalls. MS XML can always do the trick with few lines of code. You can use GET and POST as you want. Note that Text1 is a Text Box.

Code: Select all

	Dim req As XMLHTTP
	Set req = New XMLHTTP
	req.open "GET", "https://robot.lk/index.php", False
	req.send ""
	Text1.Text = req.responseText
You will have to add the following DLL from References.
MSXML.PNG
MSXML.PNG (17.96 KiB) Viewed 12045 times
User avatar
Rksk
Major
Major
Posts: 730
Joined: Thu Jan 07, 2010 4:19 pm
Location: Rathnapura, Sri Lanka

Re: How to scrap text from internet in VB6

Post by Rksk » Wed Sep 28, 2011 9:44 pm

Neo wrote:I thought you are submitting an article. Didn't thought it's a question ;)

Okay...Here is one short method. Wininet produces problems behind proxies and firewalls. MS XML can always do the trick with few lines of code. You can use GET and POST as you want. Note that Text1 is a Text Box.

Code: Select all

	Dim req As XMLHTTP
	Set req = New XMLHTTP
	req.open "GET", "https://robot.lk/index.php", False
	req.send ""
	Text1.Text = req.responseText
You will have to add the following DLL from References.
MSXML.PNG
Thank you very much neo for fast a small code. :biggrin:

:?: How use POST method?
User avatar
Enigma
Lieutenant
Lieutenant
Posts: 74
Joined: Sun Jan 16, 2011 12:40 am
Location: Colombo, Sri Lanka

Re: How to scrap text from internet in VB

Post by Enigma » Wed Sep 28, 2011 10:35 pm

Very nice Neo. Never knew that Activex control . :biggrin: :biggrin: :biggrin: . Yeah this is a faster and much more easier method.
Post Reply

Return to “Visual Basic Programming”