Beautifulsoup
** - While not new, it's a powerful Python library for web scraping that has gained renewed interest due to AI-driven data processing needs.
Last verified:
What is Beautifulsoup?
Beautiful Soup is a Python library designed for pulling data out of HTML and XML files, particularly for quick-turnaround screen scraping projects. Since 2004, it has been saving programmers hours or days of work by providing simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree. It transforms complex HTML/XML documents into a complex tree of Python objects that can be easily dissected to extract the data you need.
The library offers three powerful features: simple navigation/search/modification methods for parse trees, automatic conversion of incoming documents to Unicode and outgoing documents to UTF-8 (so you don't have to think about encodings), and the ability to sit on top of popular Python parsers like lxml and html5lib, allowing you to try different parsing strategies or trade speed for flexibility. You can tell it to find all links, find links of a specific class, find links whose URLs match a pattern, or find table headings with bold text.
Beautiful Soup is ideal for web scrapers, data scientists, Python developers working on quick-turnaround projects, and anyone trying to extract valuable data from poorly-designed websites. It parses anything you give it and handles the tree traversal automatically. Projects that would have taken hours take only minutes with Beautiful Soup. It supports Python 3.7 and greater, and is licensed under the MIT license.
The current release is Beautiful Soup 4.14.3 (November 30, 2025). It can be installed with pip install beautifulsoup4, or as python3-bs4 on Debian/Ubuntu and python3-beautifulsoup4 on Fedora. The library works with multiple parsers including Python's built-in html.parser, lxml's HTML and XML parsers, and the pure-Python html5lib parser.
Beautifulsoup pricing
Pricing model: Freemium
Free and open source under the MIT license. The library is completely free to use, and you can download the tarball and drop the bs4/ directory into any Python application without installing it. A Tidelift subscription is available for enterprise support, which supports the maintainers and provides enterprise-grade support for Beautiful Soup and other free software projects your organization depends on. There is no paid tier or premium version - all features are free.
Beautifulsoup pros
- Simple methods for navigating, searching, and modifying parse trees
- Automatic Unicode conversion for incoming documents
- Automatic UTF-8 encoding for outgoing documents
- Supports multiple parsers (html.parser, lxml, html5lib)
- Pythonic idioms make extraction code very concise
- Handles malformed HTML gracefully with browser-like heuristics
- Built-in CSS selector support via Soup Sieve library
- Multi-valued attributes like class handled automatically as lists
- find_all() and find() methods with flexible filters
- Supports searching by tag name, attributes, CSS class, and text
- Regular expression support for pattern matching
- Tree traversal methods for parents, siblings, and next/previous elements
- get_text() method easily extracts all text from a page
- MIT license allows packaging library with your application
- Saves hours or days of work on screen scraping projects
- Can pass strings, files, or URLs directly to constructor
- Tag objects can be called like functions as shorthand for find_all()
Beautifulsoup cons
- Slower than pure lxml for very large documents
- html5lib parser is very slow compared to other parsers
- External C dependency required for lxml parser
- External Python dependency required for html5lib parser
- Not as fast as lxml overall
- Less lenient than html5lib when using html.parser
- Different parsers generate different trees for invalid documents
- Python 2 support discontinued as of January 1, 2021
- Beautiful Soup 3 no longer supported as of December 31, 2020
- Cannot edit NavigableString in place, must use replace_with()
Frequently asked questions about Beautifulsoup
What is Beautiful Soup used for?
Beautiful Soup is a Python library for pulling data out of HTML and XML files. It is commonly used for screen scraping and web scraping projects to extract valuable data from websites. It parses HTML/XML documents into a tree structure that you can navigate, search, and modify to extract exactly what you need. Since 2004, it has been saving programmers hours or days of work on quick-turnaround screen scraping projects.
How do I install Beautiful Soup?
You can install Beautiful Soup 4 with pip install beautifulsoup4. On Debian and Ubuntu, it is available as the python3-bs4 package via apt-get install python3-bs4. On Fedora, it is available as python3-beautifulsoup4. You can also download the tarball and drop the bs4/ directory into your application's codebase to use it without installing, which is allowed under the MIT license.
Which parsers does Beautiful Soup support?
Beautiful Soup supports Python's built-in html.parser (batteries included, decent speed), lxml's HTML parser (very fast, external C dependency), lxml's XML parser (very fast, the only currently supported XML parser), and html5lib (extremely lenient, parses pages like a web browser, creates valid HTML5, but very slow). If you can, the documentation recommends installing and using lxml for speed.
What Python versions does Beautiful Soup support?
Beautiful Soup 4 is supported on Python versions 3.7 and greater. Support for Python 2 was discontinued on January 1, 2021—one year after the Python 2 sunsetting date. Beautiful Soup 3 does not support Python 3 and was also discontinued on January 1, 2021. If you have active projects using Beautiful Soup 3, you should migrate to Beautiful Soup 4 as part of your Python 3 conversion.
How do I search for tags by CSS class?
Since 'class' is a reserved word in Python, you cannot use it as a keyword argument. As of Beautiful Soup 4.1.2, you can search by CSS class using the keyword argument class_. For example: soup.find_all('a', class_='sister') finds all anchor tags with class 'sister'. You can also use Tag.select() with CSS selectors like soup.css.select('p.strikeout.body') to find tags matching multiple CSS classes.
What is the difference between find() and find_all()?
The find_all() method scans the entire document looking for all results that match your filters and returns a list. The find() method is used when you only want to find one result - it returns just the single result instead of a list containing it. If find_all() can't find anything, it returns an empty list. If find() can't find anything, it returns None. find() is equivalent to find_all() with limit=1, except find() returns the object directly rather than a list.
How do I extract all text from a page?
You can use the get_text() method on a BeautifulSoup object to extract all text from a page. For example: print(soup.get_text()) will print all the text content from the parsed document. This is a common task when scraping web pages and Beautiful Soup makes it very simple with this single method call.
How do I handle multi-valued attributes like class?
Beautiful Soup automatically handles multi-valued attributes. By default, it stores the values of multi-valued attributes like class as a list. For example, if you have <p class='body strikeout'>, css_soup.p['class'] returns ['body', 'strikeout']. When you turn a tag back into a string, the values are consolidated. You can force all attributes to be stored as strings by passing multi_valued_attributes=None to the BeautifulSoup constructor.
What objects does Beautiful Soup create?
Beautiful Soup transforms HTML documents into four kinds of Python objects: Tag (corresponds to an XML or HTML tag), NavigableString (contains text strings within tags), BeautifulSoup (represents the parsed document as a whole), and Comment (a special type of NavigableString for HTML comments). There are also special subclasses like Stylesheet for CSS in <style> tags, Script for JavaScript in <script> tags, and Template for HTML templates.
Where can I get help or report bugs?
If you have questions about Beautiful Soup or run into problems, you can send them to the discussion group. If you find a bug, file it on Launchpad where development happens. If it's a security vulnerability, report it confidentially through Tidelift. The documentation also notes that when reporting an error involving parsing an HTML document, you should mention something about that document to help with debugging.