342 commits

Author	SHA1	Message	Date
Terrtia	fb35be77f4	chg: [Tracker] add more info	2019-09-13 11:06:41 +02:00
Terrtia	01ff202ae8	chg: [update] add v2.2 update + fix default update + fix Empty Item.get_tags() + add new LAUNCHER options	2019-09-12 13:25:29 +02:00
Terrtia	0ced25f3be	chg: [trackers] filter trackers list by type + minor fix	2019-09-11 15:33:04 +02:00
Terrtia	bd8735a6e5	chg: [merge] master	2019-09-04 10:24:46 +02:00
Terrtia	e0bb973989	fix: [API doc] get bitcoin metadata + list of items	2019-09-04 10:06:52 +02:00
Terrtia	dfd1128daa	chg: [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail metadata + items list	2019-08-16 17:59:44 +02:00
Alexandre Dulaunoy	ab45ac0fef	Merge pull request #370 from CIRCL/api_v1 ... Api v1	2019-08-14 14:28:52 +02:00
Terrtia	7ed09bc923	chg: [UI term tracker] refractor term management: trackers list + show trackers + add new trackers	2019-08-14 09:44:49 +02:00
Terrtia	80f9535074	chg: [api] add endpoint: get tracked item_id by uuid and daterange	2019-08-09 17:07:54 +02:00
Terrtia	4b0819a644	chg: [api] add endpoint: delete tracker term (regex/set/word)	2019-08-09 15:22:57 +02:00
Terrtia	1008c7c4fe	chg: [Term Tracker] refractor term tracker word/set/regex modules + remove old modules	2019-08-09 14:20:13 +02:00
Terrtia	d9bdfecef3	fix: [Term Tracker module] chg module flow	2019-08-07 12:19:42 +02:00
Terrtia	bb6d3a6a26	chg: [Term tracker] add term tracker module (word + set) + API: add new term to track (word + set + regex)	2019-08-07 12:08:24 +02:00
Terrtia	28320a32a6	chg: [term] refractor + add new tracked word/set	2019-08-06 17:03:49 +02:00
Terrtia	e28d563523	chg: [api] use POST with parameters + add API unittest	2019-08-05 16:00:23 +02:00
Terrtia	5e1ae8a893	chg: [api] add new endpoints: get tag metadata + get all tags	2019-08-01 14:36:52 +02:00
Terrtia	98fb6ecef7	fix: [api doc] typo	2019-08-01 13:43:28 +02:00
Terrtia	4c20f58a52	chg: [api] add advanced get item via POST + use same query for each get item	2019-08-01 13:16:57 +02:00
Terrtia	8c02c1b00b	chg: [api] add 404 errors code + add future endpoints in doc	2019-08-01 09:45:59 +02:00
Terrtia	918b4c28ed	fix: [api] fix errors handler 404 405, return json	2019-07-31 13:24:43 +02:00
Terrtia	5f5e86bb13	fix: [Tags + api] fix dict keys name + fix documentation errors output	2019-07-31 11:15:34 +02:00
Terrtia	44cf5bb4af	chg: [API v1] add API documentation + update/delete items tags + Flask_tags refractor	2019-07-30 13:49:21 +02:00
Terrtia	6af9514a48	chg: [API] add GET: item metadata + item content + item tags	2019-07-26 15:44:29 +02:00
Terrtia	0a756294fe	chg: [API] import item (text)	2019-07-26 14:28:02 +02:00
Terrtia	3a8531cafa	chg: [API + import] add API format + item_import refractor	2019-07-25 17:26:32 +02:00
Terrtia	09ecc4d93f	chg: [Crawler] add default crawler config + update default user_agent	2019-07-24 10:18:10 +02:00
Terrtia	bb65179e50	fix: [user_management] fix tokens duplicate + check user_acl_integrity + add login errors messages	2019-06-20 15:49:40 +02:00
Terrtia	6eec0aabfe	Merge branch 'master' into user_management	2019-06-19 15:11:44 +02:00
Terrtia	34e57fe7af	chg: [user_managemant] clean code + check password and email length	2019-06-19 15:00:25 +02:00
Terrtia	42d32ef310	fix: [domain history] fix domain status	2019-06-18 11:19:56 +02:00
Terrtia	e1bf59aded	chg: [UI crawler, show_domain] add domain history list + navigation	2019-06-18 10:58:37 +02:00
Terrtia	b4f06c21f9	chg: [backend crawler] domains: download 1 archive by crawled (most recent)	2019-06-07 13:47:44 +02:00
Terrtia	3fe9d14e9f	chg: [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password	2019-06-06 21:27:13 +02:00
Terrtia	99e35c51ec	Merge branch 'master' into user_management	2019-06-05 16:59:36 +02:00
Terrtia	4b0a7210b8	Merge branch 'master' of https://github.com/CIRCL/AIL-framework	2019-05-24 13:27:44 +02:00
kovacsbalu	f61d830678	Use default encoding error from redis.	2019-05-16 14:24:03 +02:00
Terrtia	6480744641	chg: [PgpDump] add PgpDump backend ... TODO: UI	2019-05-14 17:49:31 +02:00
kovacsbalu	05079c143c	Fix #314 ... Replace char on redis encoding error. Try to use local file on other error.	2019-05-14 16:06:20 +02:00
Terrtia	64ff94ce5f	chg: [user_management] create + check user password	2019-05-03 16:52:05 +02:00
Terrtia	1ab1a55a4f	chg: [UI] add basic user management	2019-05-02 17:31:14 +02:00
Terrtia	a183695ddb	chg: [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags	2019-04-26 15:14:29 +02:00
Terrtia	9868833c77	chg: [crawled screenshot] use sha256 as filepath	2019-04-24 14:09:04 +02:00
Terrtia	68077316eb	chg: [git directory] add git helper	2019-04-16 17:26:35 +02:00
Terrtia	3cc614a1ad	chg: [DB] update items tags metadata	2019-04-11 11:58:06 +02:00
Terrtia	d44acea04d	chg: [merge] merge update into advanced_crawler	2019-04-10 16:41:06 +02:00
Terrtia	2589fc2161	chg: [merge] merge master into update branche	2019-04-10 15:43:15 +02:00
Terrtia	b4bee29a33	Merge branch 'master' into advanced_crawler	2019-04-10 15:37:42 +02:00
Terrtia	5165a5de2f	fix: [Crawler] fix index + redis history key	2019-03-22 17:14:27 +01:00
Terrtia	f64c385343	chg: [Crawler] handle port: crawling + history	2019-03-22 16:48:07 +01:00
Terrtia	c2bc72c72e	chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items	2019-03-21 11:11:53 +01:00
Terrtia	65e6eae00d	chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition	2019-03-19 13:34:41 +01:00
Jason Hedden	23fb91d326	Add docker-compose support ... This commit separates the AIL Framework services into individual Docker containers that can be managed with docker-compose. It's intended to ease the installation, development and troubleshooting procedures for all-in-one deployments. No changes to architecture or base code are required in this commit. Changes to existing files: .gitignore * Ignore pystemon archive directory Dockerfile: * Add AIL shell environment variables * Install pystemon and crawler pip requirements into AIL virtual environment Docker-compose notes: * All containers are participating in a service network `network_mode: service:flask`. This allows the containers to share the same IP namespace to accommodate hard-coded localhost entries. * By default persistent data is saved to the following directories in the local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS, pystemon/archives This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment would look like: ``` git clone https://github.com/CIRCL/AIL-framework.git cd AIL-framework cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg (optionally enable activate_crawler in config.cfg) docker-compose build docker-compose up -d ```	2019-03-11 19:46:34 -05:00
Terrtia	d3b3942ad7	chg: [Flask UI] refractor crawler frontend + show crawled domain	2019-03-06 15:05:58 +01:00
Terrtia	7b32d7f34e	chg: [Crawler] major refractor	2019-02-25 16:38:50 +01:00
Terrtia	da78d0552d	chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist	2019-02-19 11:41:45 +01:00
Terrtia	516238025f	chg: [Crawler] add bootsrap4 src + refractor crawler	2019-02-05 17:16:44 +01:00
Terrtia	535f44044d	fix: [Onion] add timeout signal	2019-01-29 09:46:03 +01:00
Terrtia	6ee4b7bf93	Merge branch 'master' into update	2018-12-19 15:23:34 +01:00
Terrtia	4e08aaa80f	chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains) ... todo: Shows domain history	2018-12-17 15:58:48 +01:00
Terrtia	31a8dfe0b3	fix: [AIL] use only relative paths ... pastes duplicates are fixed on the fly	2018-11-21 16:45:25 +01:00
Terrtia	cb41c54135	fix: [lib_refine] add dns configuration	2018-11-21 08:49:58 +01:00
Terrtia	d9b776d58a	merge master	2018-11-14 15:55:42 +01:00
Terrtia	b31618eb6a	chg: [Mixer] config default feeder name #291	2018-11-09 15:26:26 +01:00
Terrtia	979472df15	chg: [Terms] tag tracked terms	2018-11-06 13:38:37 +01:00
Terrtia	aa16c52cbf	fix: [NotificationHelper] add config	2018-11-05 14:20:12 +01:00
Terrtia	5872cf9196	fix: [Scripts] Remove absolute path	2018-11-02 16:07:27 +01:00
Terrtia	ea7fd8d54c	fix: [Duplicate] show duplicate with relative path	2018-10-24 15:47:03 +02:00
Terrtia	5afbc3179c	Merge branch 'master' of https://github.com/CIRCL/AIL-framework	2018-10-24 15:16:41 +02:00
Terrtia	8a6ac6be1e	fix: [Paste] create absolute path	2018-10-24 15:16:12 +02:00
Sami Mokaddem	d9bc2d6f44	Merge pull request #266 from kovacsbalu/fix-regexp-timeout ... Stop regexp processing after timeout (60sec)	2018-10-09 14:48:08 +02:00
kovacsbalu	272b0fe960	RegexpForTerm timeout from config ... Critical log removed	2018-10-09 09:32:32 +02:00
Terrtia	44d19fbb80	fix: [pystemon-feeder] paste save path	2018-10-02 16:02:59 +02:00
Alexandre Dulaunoy	59df5f0e12	chg: [config] Tor crawler disabled by default in sample configuration ... (to match HOWTO when enabling it)	2018-10-02 14:17:58 +02:00
Terrtia	04b9d9fc1d	chg: [Crawler] add docs	2018-09-27 11:14:29 +02:00
Terrtia	e4a5f66a10	Merge branch 'master' into onion_crawler	2018-09-24 16:35:04 +02:00
Terrtia	50c81773e9	chg: [Crawler] add launcher and install	2018-09-24 16:23:14 +02:00
Terrtia	ead4dba0d8	fix: [MISP feeder] typo filename-config name	2018-09-21 11:02:08 +02:00
Terrtia	6edc1ddbeb	chg: [HiddenServices] get domain related to other domains	2018-09-21 10:34:06 +02:00
Terrtia	5eadc2755f	chg: [Flask] add prefix in config to flask routes	2018-09-20 10:38:19 +02:00
Terrtia	ea3d2c1977	Merge branch 'master' into statistic	2018-09-12 10:06:53 +02:00
Terrtia	6f0817365a	chg: [Crawler UI] display domain information	2018-09-12 09:55:49 +02:00
Terrtia	ca982e13e1	chg: [Crawled Domain] show crawled pastes by domain	2018-08-27 14:34:08 +02:00
Terrtia	823c33174a	Merge branch 'master' into onion_crawler	2018-08-27 11:20:51 +02:00
Terrtia	5c902640e9	Merge branch 'master' into onion_crawler	2018-08-27 11:05:25 +02:00
Terrtia	d42dd118a4	chg: [Domain crawled] add random screenshot	2018-08-27 11:02:39 +02:00
Terrtia	a586548047	Merge branch 'master' into base64	2018-08-24 10:35:28 +02:00
Terrtia	ced0b1e350	chg: [I2P] add default config	2018-08-24 10:24:03 +02:00
Terrtia	e9580d6775	chg: [Crawler] change BDD, save i2p links	2018-08-21 15:54:53 +02:00
Terrtia	ed559d9f4a	chg: [Showpaste] add screenshot + improve onion db	2018-08-16 17:24:39 +02:00
Terrtia	8b1c10b38c	chg: [Onion] add onion splash crawler	2018-08-09 17:42:21 +02:00
Thirion Aurélien	000d5bf40a	fix: add miss config	2018-08-07 13:07:08 +02:00
Terrtia	940b96734a	Merge branch 'master' into statistic	2018-08-01 09:29:51 +02:00
Terrtia	39f54d4420	chg: [Mail] add country statistic	2018-07-30 09:21:22 +02:00
Terrtia	57b631233b	add: [BankAccount] iban detection	2018-07-26 15:31:58 +02:00
Terrtia	8beb63b6a8	fix: [Queues] module with no subscriber	2018-07-24 15:57:18 +02:00
Terrtia	8c97c70a3f	Merge branch 'master' into base64	2018-07-24 15:21:32 +02:00
Terrtia	bd5f83f0eb	chg: refractor base64 encoded to hash	2018-07-20 10:32:52 +02:00
Terrtia	9a8e37fb0b	fix: decoder bdd	2018-07-20 09:43:09 +02:00
Terrtia	ad26f016e3	add: Decoder Module, decode binary, hex and base64	2018-07-19 16:50:42 +02:00
Terrtia	e25dd52d64	add: Binary decoder	2018-07-18 11:45:19 +02:00
Terrtia	0cac3146f1	fix: remove hardcoded mail url	2018-07-17 15:11:25 +02:00
bui	bd5c42caeb	add LibInjection Module	2018-07-04 14:28:10 +02:00
Terrtia	16c700ddb4	new: add base64 display on paste + base64 statistics	2018-06-29 10:02:29 +02:00
Terrtia	af9db53171	add tag fp and tp	2018-06-19 15:09:26 +02:00
Terrtia	31fc33ab03	misp event creation, auto and manual + the hive alert auto creation, manual case creation	2018-06-14 16:51:06 +02:00
Terrtia	6143bc3dce	submit users input paste	2018-06-05 16:58:04 +02:00
Terrtia	93dcdf4f23	fix duplicate	2018-05-31 09:01:47 +02:00
Terrtia	0d164b6f4c	add tag js and css	2018-05-16 14:48:15 +02:00
Terrtia	f66a528bc2	fix Duplicate, save list of duplicates on disk + prevent empty hash creation	2018-05-15 23:28:47 +02:00
Terrtia	225fe76c96	fix duplicate + get raw paste	2018-05-11 16:07:30 +02:00
Terrtia	d7ba5533be	fix duplicate + fix issue #200	2018-05-11 13:25:45 +02:00
Terrtia	8599d7142d	python 3 upgrade instructions	2018-05-09 13:32:25 +02:00
Terrtia	142c1dc9b0	clean	2018-05-09 13:03:46 +02:00
Terrtia	3841c159a0	change leveldb to ardb	2018-05-07 14:50:40 +02:00
Terrtia	96a3df42bc	decode with redis connection	2018-05-04 13:53:29 +02:00
Terrtia	a900d5f08d	add python 2 to 3 bash + fix Duplicate + clean	2018-05-03 16:21:33 +02:00
Terrtia	c52caebe7c	improve Duplicate + tlsh + add time out handler + debug + clean	2018-05-02 17:07:10 +02:00
Terrtia	c3428df5dc	add apiKeys module	2018-04-26 14:42:39 +02:00
Terrtia	8738b7cf75	fix track term and avoid duplicate mail address	2018-04-24 16:44:37 +02:00
Thirion Aurélien	ef3c335b62	Delete config.cfg.backup	2018-04-23 14:54:23 +02:00
Terrtia	f93fe9aeb2	add bitcoin module	2018-04-23 14:51:53 +02:00
Terrtia	3fc48db903	add Base64 module	2018-04-20 10:48:44 +02:00
Terrtia	8571a86344	frontend python 3.5 upgrade	2018-04-17 16:06:32 +02:00
Terrtia	9c82dd90ec	python 3 backend upgrade	2018-04-16 14:50:04 +02:00
CD Nguyen	e9e3f17363	support non TSL SMTP servers	2018-03-30 11:35:37 +02:00
Philipp Schmied	fd329aaba9	config.cfg.sample: Merged overwritten changes from upstream	2018-02-27 16:16:57 +01:00
Philipp Schmied	5b1f0b0212	Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms.	2018-02-27 15:12:02 +01:00
Sami Mokaddem	c39e537cbe	Merge branch 'iss160' into showDiff ... Preemptively preventing merge-conflict with branch iss160 (moving to configuration)	2017-12-12 11:33:22 +01:00
Sami Mokaddem	304213cfa4	update: Possibility to change the authorized max line length in the config for the diff of duplicates	2017-12-12 11:00:26 +01:00
Sami Mokaddem	860acbf7cf	fix: Renamed Credentials into Credential sectiont title ... config change: By default, zmq listen to both local and CRF	2017-12-12 09:29:15 +01:00
Sami Mokaddem	9d0d0b4303	update: Moved filtering operation (thresholds, number of matching in the ... categ file) in the configuration file. It permits to better control the flow of pastes. Also set default mixer duplicate filtering to 3 (Do not filter)	2017-12-11 17:28:34 +01:00
Sami Mokaddem	2a967c4d92	update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment	2017-11-23 07:13:44 +01:00
Sami Mokaddem	6f0227a78a	Starting support of python3	2017-11-20 12:12:06 +01:00
Sami Mokaddem	652b00f4b2	Renamed BrowseWarningPaste into alertHandler	2017-11-15 16:15:43 +01:00
=	64ec44085d	Dynamically choose correct port for LevelDB in script	2017-08-22 17:52:15 +02:00
Mokaddem	882ab88b00	Draft module + web link	2017-07-18 16:57:15 +02:00
Mokaddem	b60e164380	Added support of onion in browseWarningPaste	2017-04-26 11:47:02 +02:00
Mokaddem	da081bae42	Merge branch 'regexInTerms' into production	2017-04-18 16:59:40 +02:00
Mokaddem	c8baabd882	Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...)	2017-04-18 15:28:21 +02:00
Mokaddem	2da4c572c7	New feature in Terms: regex and set of words support (draft)	2017-03-28 17:42:44 +02:00
Mokaddem	e225090e07	Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample	2017-03-15 16:36:51 +01:00
Mokaddem	adc37ab50e	Indexer max size now in config	2017-03-15 12:14:41 +01:00
Olivier MEDOC	16044d4d36	pystemon-feeder: add the option bind in configuration file instead of address ... This is essentially because multiple feeds can be used in the address configuration variable.	2017-01-13 14:58:18 +01:00
Olivier MEDOC	9f9c265cb0	SentimentAnalysis: use lexicon file path from the ail configuration file	2017-01-11 11:07:08 +01:00
ptitdoc	c0fac82031	Add pystemon path in configuration sample	2017-01-10 18:33:46 +01:00
Mokaddem	0e39f516a9	Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.	2017-01-09 14:12:26 +01:00
Mokaddem	e70b9cd15c	Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.	2016-12-23 10:31:26 +01:00
Mokaddem	7438f16c63	Added new config	2016-12-22 10:06:35 +01:00
Alexandre Dulaunoy	eb14005fcf	Fix: as JSON decode needs a string, the empty list is a string.	2016-08-29 12:08:12 +00:00
Alexandre Dulaunoy	3b6e3e156e	Merge branch 'master' of github.com:CIRCL/AIL-framework	2016-08-26 15:57:45 +00:00