fakefilter 1.1.13 → 1.1.1582

package/LICENSE.md

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2022, 7c
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

package/README.md

@@ -1,27 +1,77 @@
-# Fakefilter Service
+# FakeFilter
+The intention of this project is to sweep temporary and fake email addresses from registrations. We collect fake email providers and their domains and want to publish them here in different formats. We want that site owners use this repo as reliable source to find out all fake/temp provider registration attempts.
 
-this project will harvest domains of fake/temp-email providers and store them in mysql table.  This project stores everything to mysql database. The `repo/` folder is mapped to `https://github.com/7c/fakefilter` repo which is supposed to be the distribution repo
+# Why FakeFilter
+There are other lists like https://gist.github.com/adamloving/4401361 , https://disify.com/ , https://github.com/wesbos/burner-email-providers , https://github.com/disposable-email-domains/disposable-email-domains , https://github.com/elliotjreed/disposable-emails-filter-php , https://github.com/Propaganistas/Laravel-Disposable-Email , https://github.com/disposable/disposable  - they do work with submissions. This is greatly appreciated service for the community. We want to thank to them. FakeFilter has different approach: we do actively monitor all known providers and get their domains almost realtime and distribute daily. All our system is automated. Your help is still required; if you submit us new providers, we will write services to watch those providers REALTIME. Users do have rights to fake their emails, but also service providers do have the right to know about this fact. We have nothing against free/fake services, this service aims to be automated and reliable.
 
-## Testing specific provider for Development
-You may want to start every provider seperatly (for development reasons for example) just by calling `node providers/<name>.js`, this would open the puppeteer in non-headless mode. This will allow you to debug
+# Stats
+## Daily Coverage
+Amount of coverage, actions against all known Disposable Email Providers per Day
+![Monthly Coverage](https://fakefilter.net/public/img/dynamic/chart1.png)
 
-## Fetching specific provider
-`node fetch.js <providerName> [<providerName> <providerName>]`
+## Monthly Unique Disposable Domains
+Number of unique known validated Disposable Domains in Total at given Month
+![Monthly Disposable Domains](https://fakefilter.net/public/img/dynamic/chart3.png)
 
-## Repo/ folder
-this is seperate [github](https://github.com/7c/fakefilter) repo. Use `repo-update.js` from crontab to populate it. 
 
-## repo-update.sh
+## Success Rate Charts
+We monitor every disposable email provider we know about and make charts about success rate in % per day. You can access provider based success rate charts at [here](CHARTS.md)
+
+## Adding new Providers
+Please help us to add new Providers to this list, simply create an issue with the title as providers main sitedomain, this way we can group them by domain. Your help is appreciated.
+
+## Online Formats
+We have implemented a RESTful API, you can check the API at [fakefilter.net](https://fakefilter.net/static/docs/restful/). This database is aimed to serve realtime, so once we detect a new domain, this API will have it. We recommend using the Offline format as backup since we cannot guarantee the uptime.
+
+## Offline Formats
+You may use one of these formats to have access to the information flow in your favorite programming language:
+
+[Markdown](markdown/README.md)
+
+[Json](json/data.json)
+This format supports firstseen,lastseen,randomSubdomain properties
+
+[Json v2](json/data_version2.json)
+A fakedomain might belong to multiple providers, in this format we list their PSL (Public Suffix List) based domain and all known hostnames belonging to this PSL domain. Ths version also supports IDN domains, all domains are punycode.
+
+[Txt](txt/data.txt)
+
+## Expired Domains
+Our service removes domains from fakefilter (json/api/markdown/txt) after a year of inactivity. If these domains reappear, they are reinstated in our database for a minimum of 365 days. You can find the list of expired domains per provider [here](markdown/EXPIRED.md)
+
+## Whitelist
+We whitelist major well-known providers from being blocked from fakefilter, just to make sure we do not add them automatically from our system. Some providers offer @gmail<dot>com emails, adding such domains would be painful for the community.
+
+## Javascript Interface
+We have implemented npm/javascript interface for Javascript Developers as demonstration
+
+`npm i --save fakefilter`
+
+You have to make sure that this npm package is always up2date in order this static implementation to work as designed.
+
+Offline Lookup:
 ```
---dry   will not write to files
+const { isFakeDomain, isFakeEmail } = require('fakefilter')
+
+console.log(isFakeDomain('domain.com'))
+console.log(isFakeEmail('user@domain.com'))
+```
+
+
+Online Lookup:
+
+this version does not throw at all. You either get null (which indicates any error) or false or the object as reply
 ```
+## also see examples/online.js
+const { isFakeDomainOnline, isFakeEmailOnline } = require('fakefilter')
 
-## chart-update.sh
-this is responsible to draw charts using external module
+console.log(await isFakeDomainOnline('domain.com'))
+console.log(await isFakeEmailOnline('user@domain.com'))
 
+```
 
-## Docs
-docs/website are/is made with Docusaurus, the source is inside apiserver/docs/. You can use `npm run serve` to develop and `npm run build` to build. This docs page serves as the main entry point for https://fakefilter.net
 
-## events.js
-use this tool to pull events for specific provider from mysql
+Test:
+```
+node index.js
+```

package/dist/domainLookup.d.ts

@@ -0,0 +1,7 @@
+import { FakeFilterDataset } from './types';
+export declare class DomainLookup {
+    private readonly domains;
+    constructor(dataset: FakeFilterDataset);
+    match(domain: string): string | false;
+}
+export declare function getLookup(dataset: FakeFilterDataset): DomainLookup;

package/dist/domainLookup.js

@@ -0,0 +1,46 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DomainLookup = void 0;
+exports.getLookup = getLookup;
+//#endregion
+// Immutable lookup structure built once per dataset. Keys are normalized to
+// lowercase so membership and suffix checks are case-insensitive and the input
+// only needs to be normalized a single time.
+class DomainLookup {
+    constructor(dataset) {
+        this.domains = new Set();
+        for (const key of Object.keys(dataset.domains)) {
+            this.domains.add(key.toLowerCase().trim());
+        }
+    }
+    // Returns the matched fake domain (exact host or registrable suffix) or false.
+    // A Set is used instead of direct property access so reserved keys such as
+    // `constructor` or `toString` cannot produce prototype-chain false positives.
+    match(domain) {
+        const normalized = domain.toLowerCase().trim();
+        // exact match
+        if (this.domains.has(normalized))
+            return normalized;
+        // suffix match on dot boundaries: a.b.fake.com -> fake.com
+        const parts = normalized.split('.');
+        for (let i = 1; i < parts.length; i++) {
+            const suffix = parts.slice(i).join('.');
+            if (this.domains.has(suffix))
+                return suffix;
+        }
+        return false;
+    }
+}
+exports.DomainLookup = DomainLookup;
+// Datasets are cached by identity so repeated lookups reuse the same Set.
+// The bundled default dataset is a stable singleton object, so it is cached on
+// first use just like any caller-provided dataset.
+const cache = new WeakMap();
+function getLookup(dataset) {
+    let lookup = cache.get(dataset);
+    if (!lookup) {
+        lookup = new DomainLookup(dataset);
+        cache.set(dataset, lookup);
+    }
+    return lookup;
+}

package/dist/index.d.ts

@@ -0,0 +1,9 @@
+import { FakeDomainResponse, FakeFilterDataset } from './types';
+export type { DomainDetails, FakeDomainResponse } from './types';
+export declare function hostnameFromEmailAddress(email: any): string | null;
+export declare function isFakeDomain(domain: string, json?: boolean | FakeFilterDataset): string | false;
+export declare function fetch(url: string, timeout?: number, json?: boolean): Promise<any>;
+export declare function isFakeEmail(email: string, json?: boolean | FakeFilterDataset): string | false;
+export declare function isFakeEmailOnline(email: string, timeout?: number): Promise<FakeDomainResponse | null>;
+export declare function isFakeDomainOnline(domain: string | null, timeout?: number): Promise<FakeDomainResponse | null>;
+export declare function runTests(): Promise<void>;

package/dist/index.js

@@ -0,0 +1,213 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.hostnameFromEmailAddress = hostnameFromEmailAddress;
+exports.isFakeDomain = isFakeDomain;
+exports.fetch = fetch;
+exports.isFakeEmail = isFakeEmail;
+exports.isFakeEmailOnline = isFakeEmailOnline;
+exports.isFakeDomainOnline = isFakeDomainOnline;
+exports.runTests = runTests;
+/*
+
+    Basic Typescript Interface to lookup by Email or Domain
+
+    Important: you need to auto update npm package daily to make continious feed
+
+*/
+//#region imports
+const assert_1 = __importDefault(require("assert"));
+const https = __importStar(require("https"));
+const http = __importStar(require("http"));
+const data_json_1 = __importDefault(require("./json/data.json"));
+const domainLookup_1 = require("./domainLookup");
+const apiserver = 'https://fakefilter.net';
+// const apiserver = "http://127.0.0.1:5520" -- local tests
+const scheme = apiserver.search(/^https/) === 0 ? 'https' : 'http';
+function hostnameFromEmailAddress(email) {
+    if (email && typeof email === 'string' && email.search(/@/) > 0)
+        return email.split(/@/)[1];
+    return null;
+}
+function isFakeDomain(domain, json = false) {
+    const dataset = typeof json === 'boolean' ? data_json_1.default : json;
+    return (0, domainLookup_1.getLookup)(dataset).match(domain);
+}
+function fetch(url, timeout = 5000, json = true) {
+    return new Promise((resolve, reject) => {
+        try {
+            //#region response handler
+            const handler = (res) => {
+                if (res.statusCode !== 200) {
+                    res.resume();
+                    return reject('STATUSCODE');
+                }
+                let data = '';
+                res.on('data', (chunk) => { data += chunk; });
+                res.on('close', () => {
+                    if (json) {
+                        try {
+                            return resolve(JSON.parse(data));
+                        }
+                        catch (err) {
+                            return resolve(false);
+                        }
+                    }
+                    return resolve(data);
+                });
+            };
+            //#endregion
+            const request = apiserver.search(/^https/) === 0 ? https.get(url, handler) : http.get(url, handler);
+            request.setTimeout(timeout, () => {
+                request.destroy();
+                return reject('TIMEOUT');
+            });
+            request.end();
+            request.on('error', (err) => {
+                return reject(err);
+            });
+        }
+        catch (err3) {
+            // errors like ERR_INVALID_PROTOCOL is thrown here
+            reject(err3);
+        }
+    });
+}
+function isFakeEmail(email, json = false) {
+    // a null hostname is intentionally forwarded so invalid input throws, matching the original behaviour
+    return isFakeDomain(hostnameFromEmailAddress(email), json);
+}
+function isFakeEmailOnline(email, timeout = 5000) {
+    return isFakeDomainOnline(hostnameFromEmailAddress(email), timeout);
+}
+async function isFakeDomainOnline(domain, timeout = 5000) {
+    // we intentionally do not reject because we do not want to hold
+    // the process too long, null indicates error
+    try {
+        const answer = await fetch(`${apiserver}/api/is/fakedomain/${domain}`, timeout, true);
+        if (answer && answer.hasOwnProperty('retcode') && answer.retcode === 200)
+            return answer;
+    }
+    catch (err) {
+        // error returns null
+        return null;
+    }
+    // preserve original behaviour: stay pending when the lookup is not a positive hit
+    return new Promise(() => { });
+}
+async function runTests() {
+    //#region fetch tests
+    // not existing domain
+    try {
+        await fetch(`${scheme}://nonexisting${Date.now()}.com`);
+        assert_1.default.equal(false, true); // we should never reach this position
+    }
+    catch (err) {
+        assert_1.default.equal(err.code, 'ENOTFOUND');
+    }
+    // not existing url on existing domain
+    try {
+        await fetch(`${apiserver}/notexisting`);
+        assert_1.default.equal(false, true); // we should never reach this position
+    }
+    catch (err) {
+        assert_1.default.equal(err, 'STATUSCODE');
+    }
+    // invalid protocoll
+    try {
+        await fetch(`htps://fakefilter.net/notexisting`);
+        assert_1.default.equal(false, true); // we should never reach this position
+    }
+    catch (err) {
+        assert_1.default.equal(err.code, 'ERR_INVALID_PROTOCOL');
+    }
+    // invalid url
+    try {
+        await fetch(`://fakefilter.net/notexisting`);
+        assert_1.default.equal(false, true); // we should never reach this position
+    }
+    catch (err) {
+        assert_1.default.equal(err.code, 'ERR_INVALID_URL');
+    }
+    // 404
+    try {
+        await fetch(`${apiserver}/api/is/fakedomain/`);
+        assert_1.default.equal(false, true); // we should never reach this position
+    }
+    catch (err) {
+        assert_1.default.equal(err, 'STATUSCODE');
+    }
+    // non FakeDomain
+    assert_1.default.equal((await fetch(`${apiserver}/api/is/fakedomain/fakefilter.net`)).retcode, 200);
+    // FakeDomain
+    assert_1.default.equal((await fetch(`${apiserver}/api/is/fakedomain/fakefilte r.net`)).retcode, -50);
+    //#endregion
+    const json = data_json_1.default;
+    console.log(`Running tests`);
+    const all_domains = Object.keys(json.domains);
+    //#region offline + online assertions for every known domain
+    // all domains we know must be detected as FakeDomain
+    for (const domain of all_domains) {
+        // if (domain!=='www.barryogorman.com') continue
+        // console.log((domain),isFakeDomain(domain)===domain.toLowerCase())
+        // exact match
+        // console.log(`>> ${domain}`)
+        assert_1.default.notEqual(isFakeDomain(domain), false);
+        // subdomain match for example any@sub.test.com should match if test.com is part of the filter
+        assert_1.default.notEqual(isFakeDomain(`sub.${domain}`), false);
+        assert_1.default.notEqual(isFakeDomain(`sub.sub.${domain}`), false);
+        assert_1.default.notEqual(isFakeDomain(`another.sub.sub.${domain}`), false);
+        // sub$hostname should not be detected as fakedomain - if we have test.com in the filter, subtest.com should not be detected as fake
+        if (domain.search(/^[^.]+\.[^.]+$/) == 0) {
+            assert_1.default.equal(isFakeEmail(`any@sub${domain}`), false);
+            assert_1.default.equal(isFakeDomain(`sub${domain}`), false);
+        }
+        assert_1.default.notEqual(isFakeEmail(`any@${domain}`), false);
+        assert_1.default.notEqual(isFakeEmail(`any@sub.${domain}`), false);
+        // RESTFul API query (responses are guaranteed present for these mocked/online lookups)
+        assert_1.default.equal((await isFakeDomainOnline(domain)).isFakeDomain, domain);
+        assert_1.default.equal((await isFakeEmailOnline(`any@${domain}`)).isFakeDomain, domain);
+        assert_1.default.equal((await isFakeEmailOnline(`any@sub.${domain}`)).isFakeDomain, 'sub.' + domain);
+    }
+    //#endregion
+    console.log(`OK`);
+    process.exit(0);
+}
+// basic tests
+if (require.main === module)
+    runTests();