/tech/ - Technology★

>>1072440 (OP)

You can just use twitter's API to do this. It's trivial.

>>1072442

They closed that down a while ago, you now need to apply for "dev" status to get an api key

>>1072447

Every user has an API key (at least on the mobile side, but browser users probably have one too for dynamic javascript stuff). Unless they're showing HTML delivered by the server but that's generally considered a cheap hack and so big tech doesn't do it, they like to have really polished interfaces using the devices native UI toolkit.

It might be rate limited but you can always create more users. Otherwise yeah, web scraping with Selenium or maybe Beautiful Soup if the raw html delivered by the server contains enough information for your needs.

>>1072440 (OP)

If you cant figure out the API or keys as >>1072636

mentioned, then you could use the mobile site (mobile.twitter.com). With the correct cookie, it serves plain HTML.

>>1072440 (OP)

FB is the same. Get a dev API key (free), and you use that with the official API library in the language of your choice, and you can scrape most user information that the user hasn't opted out of sharing. I used this to scrape names of everyone associated with "indivisible" when that was still being hyped. I just had to rate-limit the requests per hour. I don't think it's changed.

The Cambridge Analytica "scandal" was such a joke. FB gives ANYONE, including a nobody like myself, free reign to snoop on any of its users. That's the whole business model. Every social-media service is there to abuse the trust of its users.

>>1072684

Like >>1072653

Said you can also use the mobile site for Facebook to get plain HTML. When my ex deleted her fb and removed all contact from me I made a basic script to scrape my ex's mother's friends list and then in turn scrape all of theirs, ended up with 30,000 people. Then I tried searching through the names and some facial recognition attempts but ended up having to look by hand and eventually found her under a different name.