Accessing the Internet in Python Using Urllib Library
![Accessing the Internet in Python Using Urllib Library Accessing the Internet in Python Using Urllib Library](https://www.h2kinfosys.com/blog/wp-content/uploads/2021/02/Accessing-the-Internet-in-Python-Using-Urllib-Library-Made-with-PosterMyWall.jpg)
The urllib library is a library used to access the internet and get information from websites, including their publicly available source code. You can access and get data from a website using the ult library. Data obtained are generally in JSON format, HTML, or XML. In the tutorial, you will see how to get data from a website using the urllib library. By the end of this tutorial, you will know:
- How to a send a request to a url
- How to read HTML files from a URL
- How to get response headers from a URL
Let’s jump into it.
How to Send a Request to a URL
You can send a request to a URL using the urllib.request() method. Let us see how to run the code
#import the request library from urllib import request #send a request to open the website url = request.urlopen('https://www.h2kinfosys.com/blog/') #print the result code print('The result code is ', url.getcode()) #print the status print('The status is ', url.status)
Output: The result code is 200 The status is 200
Let’s unpack the code above. We begin by importing the request function from urllib library. Afterward, parse the URL you wish to open with the urlopen() function. Finally, check whether the process was successful or not by printing the result code or status.
In both cases, the number 200 was returned. 200 is an HTTP code that shows the request was processed successfully. Another successful HTTP code is 301.
However, numbers such as 404 or 500 are error codes.
How to read HTML files from a URL
You can read HTML file from a url using the request() method. The code below reads the HTML codes for the website defined.
Output:
The output is a bunch of HTML codes.
How to get response headers from a URL
You can get the website headers using the getheaders() method. If you don’t know what a header is, the header of a website is simply the website’s metadata. The code below gets the header of the URL passed in.
#import the request library from urllib import request #send a request to open the website url = request.urlopen('https://www.h2kinfosys.com/blog/') #print the header print(url.getheaders())
Output:
[('Date', 'Sun, 07 Feb 2021 14:32:30 GMT'), ('Server', 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.5'), ('X-Powered-By', 'PHP/7.4.5'), ('Link', '<https://www.h2kinfosys.com/blog/wp-json/>; rel="https://api.w.org/"'), ('Link', '<https://www.h2kinfosys.com/blog/>; rel=shortlink'), ('Vary', 'Accept-Encoding'), ('Cache-Control', 'max-age=172800'), ('Expires', 'Tue, 09 Feb 2021 14:32:30 GMT'), ('Strict-Transport-Security', 'max-age=31536000'), ('Connection', 'close'), ('Transfer-Encoding', 'chunked'), ('Content-Type', 'text/html; charset=UTF-8')]
Note that there is a cleaner way of scraping data from a website – using Beautifulsoup and the requests library. You may decide to use the urllib library to avoid external dependencies.
One more thing. It is crucial to point out that many popular websites such as Google, Twitter, Facebook, Amazon, Wikipedia, etc. are not in support of manually requesting data from their website. They would rather have you use their API to access data as it is cleaner and frees traffic they get on their URL address. Manually scraping data over a period of time may trigger their system and have your IP blocked especially if you hit them with too many requests in a short time.
If you have any questions, feel free to leave them in the comment section, and I’d do my best to answer them.