Turn any public profile page into a structured account record — usernames, display names, bios, avatars, locations, joined-at dates, follower counts, external links, and the stable internal identifiers that uniquely pin an account across renames, redesigns, and deletions.
socid_extractor parses HTML pages and API responses from 250+ schemes and returns a flat, machine-readable dictionary of account fields. No API keys required, no headless browser — just a single function call on response text.
Why it’s useful
- Stable cross-service IDs. Get GAIA ID (Google), Facebook UID, Yandex Public ID, Instagram pk, and dozens more — values that survive username changes and let you correlate accounts across leaks, archives, and search-engine indices.
- One uniform interface. Same
extract() call for Instagram, GitHub, VK, Reddit, Substack, Bluesky, TikTok — no per-platform glue code on your side.
- Field ontology. Normalized field names across platforms (
username, fullname, created_at, is_verified, …) so downstream pipelines don’t need 130 mappings.
- Battle-tested. Powers Maigret and a number of other OSINT tools.
Installation
Python: 3.10+.
pip install socid-extractor
For a clean CLI install on a workstation:
pipx install socid-extractor
The latest development version:
pip install -U git+https://github.com/soxoj/socid-extractor.git
Quick start
As a CLI:
$ socid_extractor --url https://www.deviantart.com/muse1908
country: France
created_at: 2005-06-16 18:17:41
gender: female
username: Muse1908
website: www.patreon.com/musemercier
links: ['https://www.facebook.com/musemercier', 'https://www.instagram.com/muse.mercier/', 'https://www.patreon.com/musemercier']
tagline: Nothing worth having is easy...
As a Python library:
import requests
import socid_extractor
r = requests.get('https://www.patreon.com/annetlovart')
print(socid_extractor.extract(r.text))
# {'patreon_id': '33913189', 'patreon_username': 'annetlovart',
# 'fullname': 'Annet Lovart',
# 'links': "['https://www.facebook.com/322598031832479', ...]"}
Tip — batch runs: pass --skip-fetch-if-no-url-hint to skip the HTTP request when the URL doesn’t match any known site hint (faster, but may skip generic engines such as forum templates):
$ socid_extractor --url https://example.com/foo --skip-fetch-if-no-url-hint
Supported sites
250+ schemes, covering roughly 2,000 of the sites Maigret checks — see METHODS.md for the full list.
A non-exhaustive sample:
- Major networks: Facebook (user & group pages), Instagram, VK.com, OK.ru, Reddit, TikTok, Bluesky, Tumblr, Flickr
- Google ecosystem: Google docs/maps contributions (cookies required), Google Play, YouTube
- Mail.ru: my.mail.ru user mainpage, photo, video
- Dev / writing platforms: GitHub, Stack Overflow (HTML + API), LeetCode, Hashnode, Medium, Substack, Paragraph, WordPress.org, Virgool
- Forums (universal detectors): Discourse, MediaWiki / Fandom wikis, Mastodon
- Niche / vertical: Chess.com, Roblox, MyAnimeList, Scratch, Wikipedia, DailyMotion, SlideShare, Weebly, Calendly, Amazon Author, Boosty, Warpcast (Farcaster), Fragment (TON/Telegram), Rarible, CSSBattle, lnk.bio, Spatial, TwitchTracker, Max (max.ru)
…and many others.
For data examples, see tests/test_e2e.py; for the parsing logic, see socid_extractor/schemes.py; for the field ontology, see FIELDS.md.
Use cases
- Pivot from a profile to everything you can see. One call returns the visible info plus the hidden internal IDs the platform uses behind the scenes. Background reading: Week in OSINT — Getting a grasp on Google IDs.
- Track accounts across renames, redesigns, and deletions. Stable IDs (GAIA, FB UID, Yandex Public ID, Instagram pk, …) let you re-identify the same person even when every visible field has changed. Background: Aware Online — User IDs in social-media investigations.
- Search by cross-service UID. Once you have a stable identifier you can pivot into:
- SQL / leaked databases (forum dumps, breach data) where the UID is the join key,
- Google / Yandex / archive.org indices that captured URLs containing the UID.
- Feed downstream OSINT tooling. A normalized record is much easier to ingest than per-site scrapers — used by Maigret and similar tools for enrichment.
Commercial Use
The open-source socid_extractor is MIT-licensed and free for commercial use without restriction — but page parsers break over time as platforms change their HTML and APIs, and they need active maintenance.
For serious commercial use — a hosted extraction API, or maintenance of the parsers your product depends on — reach out: 📧 [email protected]
- Extraction API — integrate
socid_extractor into your product
SOWEL classification
Maps to the following SOWEL techniques:
- Maigret — powerful namechecker that generates a report with all available info from accounts found across 3000+ sites.
- TheScrapper — scrape emails, phone numbers, and social-media accounts from a website.
- InfoHunter — open-source OSINT tool to search, collect, and analyze information online.
- YaSeeker — gather all available information about a Yandex account by login/email.
- Marple — scrape search-engine results for a given username.
See also
- ShareTrace — the mirror image of this project:
socid_extractor reads a profile page, ShareTrace takes a share link (TikTok, Instagram, ChatGPT, Discord, Telegram and more) and recovers who created or forwarded it, decoding some share tokens offline. Nothing to install if you just want to try it: the community-run Telegram bot or web UI.
Testing
Install the test extras from pyproject.toml, then run pytest:
pip install '.[test]' # pytest, pytest-rerunfailures, pytest-xdist
python3 -m pytest tests/test_e2e.py -n 10 -k 'not cookies' -m 'not github_failed and not rate_limited'
Use pip install '.[dev]' instead if you also want flake8 / mypy / black (the full set used by CI).
Every new scheme must have an e2e test in tests/test_e2e.py hitting a real URL/API. Unit tests with inline fixtures (tests/test_socid_improvements.py) are also required but do not replace e2e coverage. See docs/testing-and-ci.md for details.
Developer documentation (architecture, modules, CI) lives in docs/.
Contributing
See the contributing guide if you want to add a new scheme or fix anything.
![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4)
评论