![]() □️ If you're wondering how we've gotten this far and haven't written any code, welcome to web scraping! Deciding how you're going to grab the data you want is actually the hard part. Meaning, I wouldn't want to select one of those obscure looking class names because they're likely coming from a CSS module or computed class name that changes every time we visit the page. ![]() It's important to be as specific as possible, but not anymore. "I want to grab the text 3,605 that lives in the anchor tag with an href of /mtliendo/followers" So now is the time to use the element selector to see if there is a way to grab what we want with basic JavaScript: In this case, no luck, and the incoming requests are polluted. Usually, in that case, I like to inspect the network tab in the Developer Tools to try and find an XHR request that serves as a JSON endpoint I can try to use. If you've ever worked with the Twitter API before, you know that it requires an approval process to use their API and has gotten more restrictive over the years. Using my twitter profile as an example, we'd want to grab my follower count as a number. To kick things off, we'll start this project with an image of the data we'd like to collect: Web scraping-or data-mining, may sound scary or dangerous, but it's really just using basic tools to access data we already have access to. If you'd rather check out the code used to scrape followers from Twitter, feel free to check out this gist I put together and adapt it to your projects □īut if you'd rather understand the process, problems, and reasoning as we create it together, keep on reading □ ![]() Note that we won't be using the Twitter API, instead, we'll be web scraping the data. The end result of this tutorial is an open API and docs page deployed on Vercel.Īs for the project itself, we'll let our users pass in a Twitter handle and we'll return that users follower count. That question is what we'll be answering in this tutorial. But what if the data we want isn't in an API, or the API isn't easily accessible? Typically, when wanting to pull in external data into an application, one would use an API.
0 Comments
Leave a Reply. |