37 Commits

Author SHA1 Message Date
Faisal Salman
e8a301265a [extensions] Create a new list for Vehicles user-agent: Volvo, Rivian, BYD 2024-12-21 12:53:06 +07:00
Faisal Salman
d024fe2006 [extensions] Fix #770 - Add new fetcher bot: Bluesky 2024-12-03 21:48:01 +07:00
Faisal Salman
fc851b40c5 Improve device detection: recognize MIUI as Xiaomi 2024-11-30 08:35:19 +07:00
Faisal Salman
c37f3863c0 [sub:extensions] Rearrange mediaplayer regexes 2024-11-26 20:13:07 +07:00
Faisal Salman
60cc84d58c [sub:extensions] Add new bots: Apache-HttpClient, go-http-client, got, GuzzleHttp, Java-http-client, libwww-perl, lua-resty-http, Needle, OkHttp, node-fetch, PHP-SOAP, PostmanRuntime, superagent 2024-11-24 12:27:08 +07:00
Faisal Salman
38baf844fc Update version to 2.0.0 2024-11-18 07:55:18 +07:00
Faisal Salman
70b3003344 [submodule:helpers] Add new method isAIBot(): detect AI bots 2024-11-16 22:14:14 +07:00
Faisal Salman
2b125c8de4 [submodule:extensions] Add new bots: AI2Bot, aiHitBot, anthropic-ai, cohere-ai, Diffbot, ImagesiftBot, magpie-crawler, Omgilibot, Screaming Frog SEO Spider, Seznambot, Teoma, Timpibot, VelenPublicWebCrawler, Vercelbot, Webzio-Extended, YouBot 2024-11-16 18:40:02 +07:00
Faisal Salman
2181559b01 [submodule:extensions] Add new email: Airmail, BlueMail, eMClient, NaverMailApp, Sparrow, Yahoo 2024-11-16 15:28:35 +07:00
Faisal Salman
14caf6d204 [submodule:extensions] Add new library: java, python-urllib, python-requests 2024-11-15 17:25:43 +07:00
Faisal Salman
4a12562946 [submodule:extensions] Add new CLI: ELinks, HTTPie 2024-11-14 22:51:41 +07:00
Faisal Salman
ee51caf422 Update version to 2.0.0-rc.3 2024-11-13 17:47:44 +07:00
Faisal Salman
cbe06acaf2 [submodule:extensions] Add email clients: Evolution, KMail, Kontact 2024-11-10 19:32:37 +07:00
Faisal Salman
a0e11b701e [submodule:extensions] Add some new bots: 360Spider, Archive.org Bots, CCBot, DataForSeoBot, DuckAssistBot, Exabot, Google Bots, Meta Bots, MojeekBot, PerplexityBot, PetalBot, TurnitinBot, Yeti, YisouSpider 2024-11-10 13:08:21 +07:00
Faisal Salman
3b3361fe53 [submodule:extensions] Breaking change: rename module to library 2024-11-10 08:16:24 +07:00
Faisal Salman
a1644b6775 Update version to 2.0.0-rc.2 2024-11-07 21:00:04 +07:00
Faisal Salman
1e0175c001 Update version to 2.0.0-rc.1 2024-10-29 22:29:51 +07:00
Faisal Salman
a4c81a5600 [submodule:extensions] Add new crawler: OAI-SearchBot (OpenAI's bot for SearchGPT) 2024-09-28 10:33:33 +07:00
Faisal Salman
f00fb3a2e2 [submodule:extensions] Restore Bots as an umbrella term for any kind of automated browsers: CLIs, Crawlers, Fetchers, and Modules 2024-09-22 13:50:00 +07:00
Faisal Salman
b1dae13245 [extension][bot] Add AhrefsBot, AhrefsSiteAudit, Dotbot, Rogerbot, Uptimerobot, Coc Coc Bot 2024-08-20 14:06:06 +07:00
Pablo Osés
68ae2a76de
adding detection of MJ12Bot and SemrushBot into extension Crawlers including testing (#738) 2024-08-14 10:46:46 +07:00
Faisal Salman
a74ace8dd1 Add Yahoo! Japan to Crawler, Fetcher, & InApp browser 2024-06-22 00:13:13 +07:00
Faisal Salman
de4978e8d9 Add mediaplayer as a new browser.type 2024-06-11 09:12:36 +07:00
Faisal Salman
6b642e2838 [Extensions submodule] Add ChatGPT-User as fetcher 2024-06-08 20:18:56 +07:00
Faisal Salman
5328642e18 Update version to 2.0.0-beta.3 2024-06-08 01:17:52 +07:00
Faisal Salman
bdcd927304 Update test for extensions 2024-06-08 00:27:28 +07:00
Faisal Salman
db3423a76c BREAKING - Remove bot type, divide as crawler / fetcher
Add new crawler: Baiduspider, DuckDuckBot, & Sogou Web Spider
Add new fetcher: Mastodon, Pinterestbot, Redditbot, LinkedInBot, Discordbot, Telegrambot, Twitterbot, Snapchat Bot, WhatsApp
2024-06-08 00:06:27 +07:00
Faisal Salman
173325faa1 Add some well-known bot user-agents: Applebot, Amazonbot, Bytespider, Claudebot, Yandexbot 2024-06-06 22:36:15 +07:00
Faisal Salman
39590f112d BREAKING CHANGE - Add new property to browser: type 2024-06-02 23:04:25 +07:00
Faisal Salman
4d950db145 Update version to 2.0.0-beta.2 2024-01-28 22:34:46 +07:00
Faisal Salman
ac282df13e Update version to 2.0.0-beta.1 2023-10-02 15:11:31 +07:00
Faisal Salman
f57f8fa1a7 Update enums & extensions 2023-09-30 14:10:53 +07:00
Faisal Salman
d168b75a3a Bump version 2.0.0-alpha.3 2023-08-17 11:29:18 +07:00
Faisal Salman
6e26e38247 [extension] Add Axios, jsdom, Scrapy. Also fixes #627 #156 #217 #227
Axios: `axios/VERSION`
https://www.zenrows.com/blog/axios-user-agent#what-is-axios-user-agent

JSDOM: `Mozilla/5.0 (${process.platform || "unknown OS"}) AppleWebKit/537.36 (KHTML, like Gecko) jsdom/${jsdomVersion}`
https://github.com/jsdom/jsdom

Scrapy: `Scrapy/VERSION (+https://scrapy.org)`
https://docs.scrapy.org/en/master/topics/settings.html
2023-08-15 11:46:31 +07:00
Faisal Salman
22eae9f70c [extensions] Add GPTBot to Bots 2023-08-07 21:39:21 +07:00
Faisal Salman
4c77c5ef21 Revive the extensive list of MediaPlayers regexes by @leofiore as an Extension
(Original commit reference: 3fa1fe9f704d716e8f57dea46bccc5724da7b952)
2023-04-24 14:58:08 +07:00
Faisal Salman
9102871dea Rearrange the structure of src folders 2023-04-21 09:47:51 +07:00