python - Web scraping program cannot find element which I can see in the browser

Question

I am trying to get the titles of the streams on https://www.twitch.tv/directory/game/Dota%202, using Requests and BeautifulSoup. I know that my search criteria are correct, yet my program does not find the elements I need.

Here is a screenshot showing the relevant part of the source code in the browser:

The HTML source as text:

<div class="tw-media-card-meta__title">
  <div class="tw-c-text-alt">
    <a class="tw-full-width tw-interactive tw-link tw-link--button tw-link--hover-underline-none tw-link--inherit" data-a-target="preview-card-title-link" href="/weplayesport_en">
      <div class="tw-align-items-start tw-flex">
        <h3 class="tw-ellipsis tw-font-size-5" title="NAVI vs HellRaisers | BO5 | ODPixel &amp; S4 | WeSave! Charity Play">NAVI vs HellRaisers | BO5 | ODPixel &amp; S4 | WeSave! Charity Play</h3>
      </div>
    </a>
  </div>
</div>

Answer 1

The element you're interested in is dynamically generated, after the initial page load, which means that your browser executed JavaScript, made other network requests, etc. in order to build the page. Requests is just an HTTP library, and as such will not do those things.

You could use a tool like Selenium, or perhaps even analyze the network traffic for the data you need and make the requests directly.