{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Scraping Website Content\n", "\n", "We can use Python core library `requests` to get the content from HTML pages. Let us understand how we can pass the html content fetched by `requests` to `BeautifulSoup`.\n", "* `requests` provides `get` funcion to which we can pass web URL. - `python_page = requests.get(python_url)`\n", "* We can access content using `python_page.content`.\n", "* We can use pass the content to BeautifulSoup and parse the HTML Tags and data for further processing." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "data": { "text/html": [ "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%HTML\n", "" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import requests" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "python_url = 'https://python.itversity.com/mastering-python.html'" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "python_page = requests.get(python_url)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "requests.models.Response" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(python_page)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "b'\\n\\n\\n\\n \\n \\n \\n Mastering Python — Mastering Python\\n \\n \\n\\n \\n \\n \\n \\n\\n \\n \\n \\n \\n\\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n\\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n\\n \\n \\n\\n\\n\\n \\n \\n \\n\\n
\\n
\\n \\n\\n\\n\\n \\n\\n\\n \\n
\\n \\n
\\n
\\n
\\n
\\n \\n \\n \\n
\\n \\n\\n \\n
\\n \\n \\n \\n \\n \\n \\n
\\n \\n
\\n \\n\\n
\\n \\n
\\n \\n open issue\\n suggest edit\\n
\\n
\\n\\n\\n \\n \\n\\n \\n\\n
\\n \\n
\\n \\n \\n \\n \\n \\n \\n
\\n
\\n\\n
\\n\\n \\n
\\n \\n
\\n Contents\\n
\\n \\n \\n
\\n
\\n
\\n
\\n \\n
\\n \\n
\\n

Mastering Python\\xc2\\xb6

\\n

This course is primarily designed to learn Python as Programming language to build web or mobile applications, data engineering applications, automation etc.

\\n
\\n

About Python\\xc2\\xb6

\\n

Python is one of the leading programming language. It is an open source database and used for different types of applications.

\\n
    \\n

  • Web Applications

    • \\n

  • Mobile Applications

    • \\n

  • Data Engineering Applications

    • \\n

  • Server Automation

    • \\n

  • Troubleshooting and Debugging

    • \\n

  • Quality Assurance

    • \\n

  • Data Science based Applications

    • \\n

  • and many many more

    • \\n
\\n

It is one of the top 3 programming languages for almost a decade now along with Java and Java Script.

\\n
\\n
\\n

Course Details\\xc2\\xb6

\\n

This course is primarily designed to go through core programming using Python. It will serve as foundation for role specific courses for different types of IT Professionals or Roles. As part of this course you will be learning the following topics under core programming using Python.

\\n
    \\n

  • Overview of GCP and Setup Ubuntu VM

    • \\n

  • Setup Postgres DB using Docker

    • \\n

  • Postgres Database Operations - Basic DDL and DML

    • \\n

  • Getting Started with Python

    • \\n

  • Basic Programming Constructs

    • \\n

  • Pre Defined Functions

    • \\n

  • User Defined Functions

    • \\n

  • Overview of Collections - list and set

    • \\n

  • Overview of Collections - dict and tuple

    • \\n

  • Manipulating Collections using loops

    • \\n

  • Overview of Map Reduce Libraries

    • \\n

  • Overview of Pandas Libraries

    • \\n

  • Database Programming - CRUD Operations

    • \\n

  • Database Programming - Batch Operations

    • \\n
\\n
\\n
\\n

Desired Audience\\xc2\\xb6

\\n

Here are the desired audience for this course.

\\n
    \\n

  • College students and entry level professionals to get hands on expertise with respect to programming using Python to be prepared for the interviews.

    • \\n

  • Experienced application developers to understand key aspects of Python Programming to build Python based Web or Mobile Applications.

    • \\n

  • Data Engineers and Data Warehouse Developers to understand key aspects of Python Programming to build batch or streaming pipelines.

    • \\n

  • Testers to improve their scripting abilities to validate data in the files tables etc.

    • \\n

  • DevOps Engineers, System Administrators, Database Administrators etc to understand Python as scripting language for the automation of day to day tasks.

    • \\n

  • Data Scientists to be proficient in Python Programming to build the models.

    • \\n
\\n
\\n

Note

\\n

This course only covers Fundamentals of Python which is useful for almost all the hands on technical roles. It does not include the role specific libraries.

\\n
\\n
\\n
\\n

Prerequisites\\xc2\\xb6

\\n

Here are the prerequisites before signing up for the course.

\\n
\\n
\\n
\\n
\\n
\\n

Logistics

\\n
    \\n

  • Computer with decent configuration

    \\n
      \\n

    • At least 4 GB RAM

      • \\n

    • 8 GB RAM is highly desired

      • \\n
    \\n
    • \\n

  • Chrome Browser

    • \\n

  • High Speed Internet

    • \\n
\\n
\\n
\\n
\\n
\\n
\\n
\\n

Desired Skills

\\n
    \\n

  • Engineering or Science Degree

    • \\n

  • Ability to use computer

    • \\n

  • Knowledge or working experience with databases is highly desired

    • \\n
\\n
\\n
\\n
\\n
\\n
\\n
\\n
\\n

Key Objectives\\xc2\\xb6

\\n

The course is designed for the professionals to achieve these key objectives related to programming using Python as Programming Language.

\\n
\\n

Attention

\\n

This course is primarily designed to gain key database skills for application developers, data engineers, testers, business analysts etc.

\\n
\\n
\\n
\\n

Training Approach\\xc2\\xb6

\\n

Here are the details related to the training approach.

\\n
    \\n

  • It is self paced with reference material, code snippets and videos.

    • \\n

  • One can either use environment provided by us or setup their own environment using Docker.

    • \\n

  • Modules will be published as and when they are ready. We would recommend to complete 2 modules every week by spending 4 to 5 hours per week.

    • \\n

  • It is highly recommended to take care of the exercises at the end to ensure that you are able to meet all the key objectives for each module.

    • \\n

  • Support will be provided either through chat or email.

    • \\n

  • For those who signed up, we will have weekly monitoring and review sessions to keep track of the progress.

    • \\n
\\n
\\n

Attention

\\n

Spend 4 to 5 hours per week up to 8 weeks and complete all the exercises to get best out of this course.

\\n
\\n
\\n
\\n

Self Evaluation\\xc2\\xb6

\\n

The course is designed in such a way that one can self evaluate through the course and confirm whether the skills are acquired.

\\n
    \\n

  • Here is the approach we recommend you to take this course.

    \\n
      \\n

    • Go through the consolidated exercises and see if you are able to solve the problems or not.

      • \\n

    • Make sure to follow the order we have defined as part of the course.

      • \\n

    • After each and every section or module, make sure to solve the exercises. We have provided enough information to validate the output of your queries.

      • \\n

    • After the completion of the course try to solve the exercises using consolidated list.

      • \\n

    • Keep in mind that you will be reviewing the same exercises before the course, during the course as well as at the end of the course.

      • \\n
    \\n
    • \\n

  • By the end of the course, if you are able to solve the problems, then you can come to a conclusion that you are able to master the key skill called as SQL.

    • \\n
\\n
\\n
\\n
\\n
\\n\\n \\n \\n\\n
\\n \\n
\\n
\\n \\n \\n
\\n \\n Overview of Windows Operating System\\n\\n
\\n
\\n
\\n

\\n \\n By Durga Gadiraju
\\n \\n © Copyright ITVersity, Inc.
\\n

\\n
\\n
\\n
\\n\\n\\n
\\n
\\n\\n \\n \\n\\n\\n \\n \\n \\n \\n \\n \\n \\n'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# python_page.content" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "bytes" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(python_page.content)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Processing HTML Content\n", "\n", "We can pass the content and extract HTML Tags as well as data using BeautifulSoup.\n", "* We have to pass the content using `html.parser` and build the BeautifulSoup object.\n", "* Let us prettify and print the content." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from bs4 import BeautifulSoup" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " \n", " \n", " \n", " \n", " Mastering Python — Mastering Python\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", " \n", " Contents\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

\n", " Mastering Python\n", " \n", " ¶\n", " \n", "

\n", "

\n", " This course is primarily designed to learn Python as Programming language to build web or mobile applications, data engineering applications, automation etc.\n", "

\n", "
\n", "

\n", " About Python\n", " \n", " ¶\n", " \n", "

\n", "

\n", " Python is one of the leading programming language. It is an open source database and used for different types of applications.\n", "

\n", "
    \n", "
  • \n", "

    \n", " Web Applications\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Mobile Applications\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Data Engineering Applications\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Server Automation\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Troubleshooting and Debugging\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Quality Assurance\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Data Science based Applications\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " and many many more\n", "

    \n", "
    • \n", "
\n", "

\n", " It is one of the top 3 programming languages for almost a decade now along with Java and Java Script.\n", "

\n", "
\n", "
\n", "

\n", " Course Details\n", " \n", " ¶\n", " \n", "

\n", "

\n", " This course is primarily designed to go through core programming using Python. It will serve as foundation for role specific courses for different types of IT Professionals or Roles. As part of this course you will be learning the following topics under core programming using Python.\n", "

\n", "
    \n", "
  • \n", "

    \n", " Overview of GCP and Setup Ubuntu VM\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Setup Postgres DB using Docker\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Postgres Database Operations - Basic DDL and DML\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Getting Started with Python\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Basic Programming Constructs\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Pre Defined Functions\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " User Defined Functions\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Overview of Collections - list and set\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Overview of Collections - dict and tuple\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Manipulating Collections using loops\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Overview of Map Reduce Libraries\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Overview of Pandas Libraries\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Database Programming - CRUD Operations\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Database Programming - Batch Operations\n", "

    \n", "
    • \n", "
\n", "
\n", "
\n", "

\n", " Desired Audience\n", " \n", " ¶\n", " \n", "

\n", "

\n", " Here are the desired audience for this course.\n", "

\n", "
    \n", "
  • \n", "

    \n", " College students and entry level professionals to get hands on expertise with respect to programming using Python to be prepared for the interviews.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Experienced application developers to understand key aspects of Python Programming to build Python based Web or Mobile Applications.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Data Engineers and Data Warehouse Developers to understand key aspects of Python Programming to build batch or streaming pipelines.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Testers to improve their scripting abilities to validate data in the files tables etc.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " DevOps Engineers, System Administrators, Database Administrators etc to understand Python as scripting language for the automation of day to day tasks.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Data Scientists to be proficient in Python Programming to build the models.\n", "

    \n", "
    • \n", "
\n", "
\n", "

\n", " Note\n", "

\n", "

\n", " This course only covers Fundamentals of Python which is useful for almost all the hands on technical roles. It does not include the role specific libraries.\n", "

\n", "
\n", "
\n", "
\n", "

\n", " Prerequisites\n", " \n", " ¶\n", " \n", "

\n", "

\n", " Here are the prerequisites before signing up for the course.\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

\n", " \n", " Logistics\n", " \n", "

\n", "
    \n", "
  • \n", "

    \n", " Computer with decent configuration\n", "

    \n", "
      \n", "
    • \n", "

      \n", " At least 4 GB RAM\n", "

      \n", "
      • \n", "
    • \n", "

      \n", " 8 GB RAM is highly desired\n", "

      \n", "
      • \n", "
    \n", "
    • \n", "
  • \n", "

    \n", " Chrome Browser\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " High Speed Internet\n", "

    \n", "
    • \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

\n", " \n", " Desired Skills\n", " \n", "

\n", "
    \n", "
  • \n", "

    \n", " Engineering or Science Degree\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Ability to use computer\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Knowledge or working experience with databases is highly desired\n", "

    \n", "
    • \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

\n", " Key Objectives\n", " \n", " ¶\n", " \n", "

\n", "

\n", " The course is designed for the professionals to achieve these key objectives related to programming using Python as Programming Language.\n", "

\n", "
\n", "

\n", " Attention\n", "

\n", "

\n", " This course is primarily designed to gain key database skills for application developers, data engineers, testers, business analysts etc.\n", "

\n", "
\n", "
\n", "
\n", "

\n", " Training Approach\n", " \n", " ¶\n", " \n", "

\n", "

\n", " Here are the details related to the training approach.\n", "

\n", "
    \n", "
  • \n", "

    \n", " It is self paced with reference material, code snippets and videos.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " One can either use environment provided by us or setup their own environment using Docker.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Modules will be published as and when they are ready. We would recommend to complete\n", " \n", " 2 modules every week\n", " \n", " by spending\n", " \n", " 4 to 5 hours per week\n", " \n", " .\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " It is highly recommended to take care of the exercises at the end to ensure that you are able to meet all the key objectives for each module.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " Support will be provided either through chat or email.\n", "

    \n", "
    • \n", "
  • \n", "

    \n", " For those who signed up, we will have weekly monitoring and review sessions to keep track of the progress.\n", "

    \n", "
    • \n", "
\n", "
\n", "

\n", " Attention\n", "

\n", "

\n", " Spend 4 to 5 hours per week up to 8 weeks and complete all the exercises to get best out of this course.\n", "

\n", "
\n", "
\n", "
\n", "

\n", " Self Evaluation\n", " \n", " ¶\n", " \n", "

\n", "

\n", " The course is designed in such a way that one can self evaluate through the course and confirm whether the skills are acquired.\n", "

\n", "
    \n", "
  • \n", "

    \n", " Here is the approach we recommend you to take this course.\n", "

    \n", "
      \n", "
    • \n", "

      \n", " Go through the consolidated exercises and see if you are able to solve the problems or not.\n", "

      \n", "
      • \n", "
    • \n", "

      \n", " Make sure to follow the order we have defined as part of the course.\n", "

      \n", "
      • \n", "
    • \n", "

      \n", " After each and every section or module, make sure to solve the exercises. We have provided enough information to validate the output of your queries.\n", "

      \n", "
      • \n", "
    • \n", "

      \n", " After the completion of the course try to solve the exercises using consolidated list.\n", "

      \n", "
      • \n", "
    • \n", "

      \n", " Keep in mind that you will be reviewing the same exercises before the course, during the course as well as at the end of the course.\n", "

      \n", "
      • \n", "
    \n", "
    • \n", "
  • \n", "

    \n", " By the end of the course, if you are able to solve the problems, then you can come to a conclusion that you are able to master the key skill called as SQL.\n", "

    \n", "
    • \n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", " Overview of Windows Operating System\n", " \n", "
\n", "
\n", "
\n", "

\n", " By Durga Gadiraju\n", "
\n", " © Copyright ITVersity, Inc.\n", "
\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ] } ], "source": [ "soup = BeautifulSoup(python_page.content, 'html.parser')\n", "# print(soup.prettify())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 4 }