diff --git a/.gitignore b/.gitignore index 785419e5..28ed8063 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,8 @@ indexdir/ logs/ old/ +DEFAULT_PASSWORD + # Webstuff var/www/static/ !var/www/static/css/dygraph_gallery.css diff --git a/OVERVIEW.md b/OVERVIEW.md index 6378e631..38ac7e7f 100644 --- a/OVERVIEW.md +++ b/OVERVIEW.md @@ -30,6 +30,14 @@ Redis and ARDB overview # Database Map: +### Redis cache + +##### Brute force protection: +| Set Key | Value | +| ------ | ------ | +| failed_login_ip:**ip** | **nb login failed** | TTL +| failed_login_user_id:**user_id** | **nb login failed** | TTL + ## DB0 - Core: ##### Update keys: @@ -42,8 +50,6 @@ Redis and ARDB overview | | **background update name** | | | **...** | | | | -| ail:update_date_v1.5 | **update date** | -| | | | ail:update_error | **update message error** | | | | | ail:update_in_progress | **update version in progress** | @@ -52,6 +58,30 @@ Redis and ARDB overview | ail:current_background_script | **name of the background script currently executed** | | ail:current_background_script_stat | **progress in % of the background script** | +| Hset Key | Field | Value | +| ------ | ------ | ------ | +| ail:update_date | **update tag** | **update date** | + +##### User Management: +| Hset Key | Field | Value | +| ------ | ------ | ------ | +| user:all | **user id** | **password hash** | +| | | | +| user:tokens | **token** | **user id** | +| | | | +| user_metadata:**user id** | token | **token** | +| | change_passwd | **boolean** | +| | role | **role** | + +| Set Key | Value | +| ------ | ------ | +| user_role:**role** | **user id** | + + +| Zrank Key | Field | Value | +| ------ | ------ | ------ | +| ail:all_role | **role** | **int, role priority (1=admin)** | + ## DB2 - TermFreq: ##### Set: diff --git a/README.md b/README.md index eea6510f..65ed67f6 100644 --- a/README.md +++ b/README.md @@ -78,14 +78,13 @@ Type these command lines for a fully automated installation and start AIL framew git clone https://github.com/CIRCL/AIL-framework.git cd AIL-framework ./installing_deps.sh + cd ~/AIL-framework/ -. ./AILENV/bin/activate cd bin/ -./LAUNCH.sh +./LAUNCH.sh -l ``` -The default [installing_deps.sh](./installing_deps.sh) is for Debian and Ubuntu based distributions. For Arch -linux based distributions, you can replace it with [installing_deps_archlinux.sh](./installing_deps_archlinux.sh). +The default [installing_deps.sh](./installing_deps.sh) is for Debian and Ubuntu based distributions. There is also a [Travis file](.travis.yml) used for automating the installation that can be used to build and install AIL on other systems. @@ -143,23 +142,12 @@ Install using Ansible Please check the [Ansible readme](ansible/README.md). -Starting AIL web interface +Starting AIL -------------------------- -To start the web interface, you first need to fetch the required JavaScript/CSS files: - ```bash -cd $AILENV -cd var/www/ -bash update_thirdparty.sh -``` - -and then you can start the web interface python script: - -```bash -cd $AILENV -cd var/www/ -./Flask_server.py +cd bin/ +./LAUNCH -l ``` Eventually you can browse the status of the AIL framework website at the following URL: @@ -168,6 +156,8 @@ Eventually you can browse the status of the AIL framework website at the followi http://localhost:7000/ ``` +The default credentials for the web interface are located in ``DEFAULT_PASSWORD``. This file is removed when you change your password. + Training -------- diff --git a/bin/helper/gen_cert/README.md b/bin/helper/gen_cert/README.md new file mode 100644 index 00000000..d007add5 --- /dev/null +++ b/bin/helper/gen_cert/README.md @@ -0,0 +1,14 @@ +Usage +===== +These scripts are useful to generate the server's self-signed certificate. +## Root Certification Authority ## +`gen_root.sh` +This will create a rootCA.crt to verify the server's cert. +## Server certificate ## +### Configuration Files ### +Look into these files if you plan to verify the certificate: + * san.cnf holds basic information about the certificate. + * ext.3 holds in particular the 'subjectAltNames** option that is for the verification to succeed. + + ### Generation ### +`gen_root.sh` diff --git a/bin/helper/gen_cert/ext3.cnf b/bin/helper/gen_cert/ext3.cnf new file mode 100644 index 00000000..8f36435b --- /dev/null +++ b/bin/helper/gen_cert/ext3.cnf @@ -0,0 +1,5 @@ +keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment +subjectAltName = @alt_names + +[alt_names] +IP.1 = 127.0.0.1 diff --git a/bin/helper/gen_cert/gen_cert.sh b/bin/helper/gen_cert/gen_cert.sh new file mode 100755 index 00000000..f22de1cb --- /dev/null +++ b/bin/helper/gen_cert/gen_cert.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Create Server key +openssl genrsa -out server.key 4096 +# Create the Server Signing Request - non interactive, config in san.cnf +openssl req -sha256 -new -key server.key -out server.csr -config san.cnf +# Create the server certificate by rootCA, with ext3 subjectAltName in ext3.cnf +openssl x509 -req -in server.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out server.crt -days 500 -sha256 -extfile ext3.cnf +# Concat in pem +cat server.crt server.key > server.pem diff --git a/bin/helper/gen_cert/gen_root.sh b/bin/helper/gen_cert/gen_root.sh new file mode 100755 index 00000000..1f6e582b --- /dev/null +++ b/bin/helper/gen_cert/gen_root.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +# Create Root key +openssl genrsa -out rootCA.key 4096 +# Create and Sign the Root CA Certificate +openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt -config san.cnf diff --git a/bin/helper/gen_cert/san.cnf b/bin/helper/gen_cert/san.cnf new file mode 100644 index 00000000..61d8a11a --- /dev/null +++ b/bin/helper/gen_cert/san.cnf @@ -0,0 +1,11 @@ +[req] +default_bits = 4096 +prompt = no +default_md = sha256 +distinguished_name = dn + +[dn] +C=LU +L=Luxembourg +O=AIL-Default +CN = localhost diff --git a/bin/packages/User.py b/bin/packages/User.py new file mode 100755 index 00000000..829e4205 --- /dev/null +++ b/bin/packages/User.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +# -*-coding:UTF-8 -* + +import os +import redis +import bcrypt +import configparser + +from flask_login import UserMixin + +class User(UserMixin): + + def __init__(self, id): + + configfile = os.path.join(os.environ['AIL_BIN'], 'packages/config.cfg') + if not os.path.exists(configfile): + raise Exception('Unable to find the configuration file. \ + Did you set environment variables? \ + Or activate the virtualenv.') + + cfg = configparser.ConfigParser() + cfg.read(configfile) + + self.r_serv_db = redis.StrictRedis( + host=cfg.get("ARDB_DB", "host"), + port=cfg.getint("ARDB_DB", "port"), + db=cfg.getint("ARDB_DB", "db"), + decode_responses=True) + + if self.r_serv_db.hexists('user:all', id): + self.id = id + else: + self.id = "__anonymous__" + + # return True or False + #def is_authenticated(): + + # return True or False + #def is_anonymous(): + + @classmethod + def get(self_class, id): + return self_class(id) + + def user_is_anonymous(self): + if self.id == "__anonymous__": + return True + else: + return False + + def check_password(self, password): + if self.user_is_anonymous(): + return False + + password = password.encode() + hashed_password = self.r_serv_db.hget('user:all', self.id).encode() + if bcrypt.checkpw(password, hashed_password): + return True + else: + return False + + def request_password_change(self): + if self.r_serv_db.hget('user_metadata:{}'.format(self.id), 'change_passwd') == 'True': + return True + else: + return False + + def is_in_role(self, role): + if self.r_serv_db.sismember('user_role:{}'.format(role), self.id): + return True + else: + return False diff --git a/installing_deps.sh b/installing_deps.sh index e8de3fd6..99325f82 100755 --- a/installing_deps.sh +++ b/installing_deps.sh @@ -99,7 +99,17 @@ if [ -z "$VIRTUAL_ENV" ]; then fi -pushd var/www/ +pushd ${AIL_BIN}helper/gen_cert +./gen_root.sh +wait +./gen_cert.sh +wait +popd + +cp ${AIL_BIN}helper/gen_cert/server.crt ${AIL_FLASK}server.crt +cp ${AIL_BIN}helper/gen_cert/server.key ${AIL_FLASK}server.key + +pushd ${AIL_FLASK} ./update_thirdparty.sh popd @@ -123,6 +133,26 @@ python3 setup.py install HOME=$(pwd) python3 -m textblob.download_corpora python3 -m nltk.downloader vader_lexicon python3 -m nltk.downloader punkt +popd #Create the file all_module and update the graph in doc $AIL_HOME/doc/generate_modules_data_flow_graph.sh + +#### DB SETUP #### + +# init update version +git describe --abbrev=0 --tags | tr -d '\n' > ${AIL_HOME}/update/current_version + +# LAUNCH ARDB +bash ${AIL_BIN}LAUNCH.sh -lav & +wait +echo "" + +# create default user +pushd ${AIL_FLASK} +python3 create_default_user.py +popd + +bash ${AIL_BIN}LAUNCH.sh -k & +wait +echo "" diff --git a/installing_deps_archlinux.sh b/installing_deps_archlinux.sh deleted file mode 100755 index 38ac4ca7..00000000 --- a/installing_deps_archlinux.sh +++ /dev/null @@ -1,108 +0,0 @@ -#!/bin/bash - - -echo "Currently unmaintained, continue at your own risk of not having a working AIL at the end :( Will be merged into main install deps later on." -exit 1 - -set -e -set -x - -sudo pacman -Syu - -sudo pacman -S python2-pip screen gcc unzip freetype2 python2 git --needed -sudo yaourt -S snappy --needed -sudo pip2 install virtualenv - -#Needed for bloom filters -sudo pacman -S openssl python2-numpy --needed - -# DNS deps -sudo pacman -S adns --needed - -#Needed for redis-lvlDB -sudo pacman -S libev gmp --needed - -#needed for mathplotlib -test ! -L /usr/include/ft2build.h && sudo ln -s freetype2/ft2build.h /usr/include/ -sudo easy_install-2.7 -U distribute - -# REDIS # -test ! -d redis/ && git clone https://github.com/antirez/redis.git -pushd redis/ -git checkout 2.8 -make -popd - -# REDIS LEVEL DB # -test ! -d redis-leveldb/ && git clone https://github.com/KDr2/redis-leveldb.git -pushd redis-leveldb/ -git submodule init -git submodule update -make -popd - -# Faup -test ! -d faup/ && git clone https://github.com/stricaud/faup.git -pushd faup/ -test ! -d build && mkdir build -cd build -cmake .. && make -sudo make install -echo '/usr/local/lib' | sudo tee -a /etc/ld.so.conf.d/faup.conf -sudo ldconfig -popd - -# tlsh -test ! -d tlsh && git clone git://github.com/trendmicro/tlsh.git -pushd tlsh/ -./make.sh -pushd build/release/ -sudo make install -sudo ldconfig -popd -popd - - - -if [ ! -f bin/packages/config.cfg ]; then - cp bin/packages/config.cfg.sample bin/packages/config.cfg -fi - -pushd var/www/ -./update_thirdparty.sh -popd - -virtualenv AILENV - -echo export AIL_HOME=$(pwd) >> ./AILENV/bin/activate -echo export AIL_BIN=$(pwd)/bin/ >> ./AILENV/bin/activate -echo export AIL_FLASK=$(pwd)/var/www/ >> ./AILENV/bin/activate -echo export AIL_REDIS=$(pwd)/redis/src/ >> ./AILENV/bin/activate -echo export AIL_LEVELDB=$(pwd)/redis-leveldb/ >> ./AILENV/bin/activate - -. ./AILENV/bin/activate - -mkdir -p $AIL_HOME/{PASTES,Blooms,dumps} -mkdir -p $AIL_HOME/LEVEL_DB_DATA/2017 -mkdir -p $AIL_HOME/LEVEL_DB_DATA/3017 - -pip install -U pip -pip install -U -r pip_packages_requirement.txt - -# Pyfaup -pushd faup/src/lib/bindings/python/ -python setup.py install -popd - -# Py tlsh -pushd tlsh/py_ext -python setup.py build -python setup.py install - -# Download the necessary NLTK corpora and sentiment vader -HOME=$(pwd) python -m textblob.download_corpora -python -m nltk.downloader vader_lexicon -python -m nltk.downloader punkt - -#Create the file all_module and update the graph in doc -$AIL_HOME/doc/generate_modules_data_flow_graph.sh diff --git a/requirements.txt b/requirements.txt index fe98901d..fdccf9bb 100644 --- a/requirements.txt +++ b/requirements.txt @@ -43,9 +43,12 @@ psutil phonenumbers ipython -flask texttable +flask +flask-login +bcrypt + #DomainClassifier DomainClassifier #Indexer requirements diff --git a/update/v2.0/Update.py b/update/v2.0/Update.py new file mode 100755 index 00000000..4e026e1f --- /dev/null +++ b/update/v2.0/Update.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +# -*-coding:UTF-8 -* + +import os +import sys +import time +import redis +import datetime +import configparser + +if __name__ == '__main__': + + start_deb = time.time() + + configfile = os.path.join(os.environ['AIL_BIN'], 'packages/config.cfg') + if not os.path.exists(configfile): + raise Exception('Unable to find the configuration file. \ + Did you set environment variables? \ + Or activate the virtualenv.') + cfg = configparser.ConfigParser() + cfg.read(configfile) + + r_serv = redis.StrictRedis( + host=cfg.get("ARDB_DB", "host"), + port=cfg.getint("ARDB_DB", "port"), + db=cfg.getint("ARDB_DB", "db"), + decode_responses=True) + + #Set current ail version + r_serv.set('ail:version', 'v2.0') + + # use new update_date format + date_tag_to_replace = ['v1.5', 'v1.7'] + for tag in date_tag_to_replace: + if r_serv.exists('ail:update_date_{}'.format(tag)): + date_tag = r_serv.get('ail:update_date_{}'.format(tag)) + r_serv.hset('ail:update_date', tag, date_tag) + r_serv.delete('ail:update_date_{}'.format(tag)) + + #Set current ail version + r_serv.hset('ail:update_date', 'v2.0', datetime.datetime.now().strftime("%Y%m%d")) diff --git a/update/v2.0/Update.sh b/update/v2.0/Update.sh new file mode 100755 index 00000000..6eccb0e3 --- /dev/null +++ b/update/v2.0/Update.sh @@ -0,0 +1,75 @@ +#!/bin/bash + +[ -z "$AIL_HOME" ] && echo "Needs the env var AIL_HOME. Run the script from the virtual environment." && exit 1; +[ -z "$AIL_REDIS" ] && echo "Needs the env var AIL_REDIS. Run the script from the virtual environment." && exit 1; +[ -z "$AIL_ARDB" ] && echo "Needs the env var AIL_ARDB. Run the script from the virtual environment." && exit 1; +[ -z "$AIL_BIN" ] && echo "Needs the env var AIL_ARDB. Run the script from the virtual environment." && exit 1; +[ -z "$AIL_FLASK" ] && echo "Needs the env var AIL_FLASK. Run the script from the virtual environment." && exit 1; + +export PATH=$AIL_HOME:$PATH +export PATH=$AIL_REDIS:$PATH +export PATH=$AIL_ARDB:$PATH +export PATH=$AIL_BIN:$PATH +export PATH=$AIL_FLASK:$PATH + +GREEN="\\033[1;32m" +DEFAULT="\\033[0;39m" + +echo -e $GREEN"Shutting down AIL ..."$DEFAULT +bash ${AIL_BIN}/LAUNCH.sh -k +wait + +echo "" +echo -e $GREEN"Create Self-Signed Certificate"$DEFAULT +echo "" +pushd ${AIL_BIN}/helper/gen_cert +bash gen_root.sh +wait +bash gen_cert.sh +wait +popd + +cp ${AIL_BIN}/helper/gen_cert/server.crt ${AIL_FLASK}/server.crt +cp ${AIL_BIN}/helper/gen_cert/server.key ${AIL_FLASK}/server.key + +echo "" +echo -e $GREEN"Update requirement"$DEFAULT +echo "" +pip3 install flask-login +wait +echo "" +pip3 install bcrypt +wait +echo "" +echo "" + +bash ${AIL_BIN}/LAUNCH.sh -lav & +wait +echo "" + +echo "" +echo -e $GREEN"Updating AIL VERSION ..."$DEFAULT +echo "" +python ${AIL_HOME}/update/v2.0/Update.py +wait +echo "" +echo "" + +echo "" +echo -e $GREEN"Update thirdparty ..."$DEFAULT +bash ${AIL_BIN}/LAUNCH.sh -t +wait +echo "" + +echo "" +echo -e $GREEN"Create Default User"$DEFAULT +echo "" +python3 ${AIL_FLASK}create_default_user.py + + +echo "" +echo -e $GREEN"Shutting down ARDB ..."$DEFAULT +bash ${AIL_BIN}/LAUNCH.sh -k +wait + +exit 0 diff --git a/var/www/Flask_server.py b/var/www/Flask_server.py index ff6d5bac..8ba4526e 100755 --- a/var/www/Flask_server.py +++ b/var/www/Flask_server.py @@ -1,28 +1,42 @@ #!/usr/bin/env python3 # -*-coding:UTF-8 -* -import redis -import configparser -import json -import datetime +import os +import re +import sys +import ssl import time -import calendar -from flask import Flask, render_template, jsonify, request, Request + +import redis +import random +import logging +import logging.handlers +import configparser + +from flask import Flask, render_template, jsonify, request, Request, session, redirect, url_for +from flask_login import LoginManager, current_user, login_user, logout_user, login_required + +import bcrypt + import flask import importlib -import os from os.path import join -import sys sys.path.append(os.path.join(os.environ['AIL_BIN'], 'packages/')) sys.path.append('./modules/') import Paste from Date import Date +from User import User + from pytaxonomies import Taxonomies # Import config import Flask_config +# Import Role_Manager +from Role_Manager import create_user_db, check_password_strength, check_user_role_integrity +from Role_Manager import login_admin, login_analyst + # CONFIG # cfg = Flask_config.cfg baseUrl = cfg.get("Flask", "baseurl") @@ -30,10 +44,63 @@ baseUrl = baseUrl.replace('/', '') if baseUrl != '': baseUrl = '/'+baseUrl +# ========= REDIS =========# +r_serv_db = redis.StrictRedis( + host=cfg.get("ARDB_DB", "host"), + port=cfg.getint("ARDB_DB", "port"), + db=cfg.getint("ARDB_DB", "db"), + decode_responses=True) +r_serv_tags = redis.StrictRedis( + host=cfg.get("ARDB_Tags", "host"), + port=cfg.getint("ARDB_Tags", "port"), + db=cfg.getint("ARDB_Tags", "db"), + decode_responses=True) + +r_cache = redis.StrictRedis( + host=cfg.get("Redis_Cache", "host"), + port=cfg.getint("Redis_Cache", "port"), + db=cfg.getint("Redis_Cache", "db"), + decode_responses=True) + +# logs +log_dir = os.path.join(os.environ['AIL_HOME'], 'logs') +if not os.path.isdir(log_dir): + os.makedirs(logs_dir) + +log_filename = os.path.join(log_dir, 'flask_server.logs') +logger = logging.getLogger() +formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s') +handler_log = logging.handlers.TimedRotatingFileHandler(log_filename, when="midnight", interval=1) +handler_log.suffix = '%Y-%m-%d.log' +handler_log.setFormatter(formatter) +handler_log.setLevel(30) +logger.addHandler(handler_log) +logger.setLevel(30) + +# ========= =========# + +# ========= TLS =========# +ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2) +ssl_context.load_cert_chain(certfile='server.crt', keyfile='server.key') +#print(ssl_context.get_ciphers()) +# ========= =========# + Flask_config.app = Flask(__name__, static_url_path=baseUrl+'/static/') app = Flask_config.app app.config['MAX_CONTENT_LENGTH'] = 900 * 1024 * 1024 +# ========= session ======== +app.secret_key = str(random.getrandbits(256)) +login_manager = LoginManager() +login_manager.login_view = 'login' +login_manager.init_app(app) + +# ========= LOGIN MANAGER ======== + +@login_manager.user_loader +def load_user(user_id): + return User.get(user_id) + # ========= HEADER GENERATION ======== # Get headers items that should be ignored (not displayed) @@ -95,7 +162,6 @@ modified_header = modified_header.replace('', '\n'.join(to_add with open('templates/header.html', 'w') as f: f.write(modified_header) - # ========= JINJA2 FUNCTIONS ======== def list_len(s): return len(s) @@ -113,18 +179,123 @@ def add_header(response): response.headers['Cache-Control'] = 'public, max-age=0' return response +# @app.route('/test', methods=['GET']) +# def test(): +# for rule in app.url_map.iter_rules(): +# print(rule) +# return 'o' + # ========== ROUTES ============ +@app.route('/login', methods=['POST', 'GET']) +def login(): + + current_ip = request.remote_addr + login_failed_ip = r_cache.get('failed_login_ip:{}'.format(current_ip)) + + # brute force by ip + if login_failed_ip: + login_failed_ip = int(login_failed_ip) + if login_failed_ip >= 5: + error = 'Max Connection Attempts reached, Please wait {}s'.format(r_cache.ttl('failed_login_ip:{}'.format(current_ip))) + return render_template("login.html", error=error) + + if request.method == 'POST': + username = request.form.get('username') + password = request.form.get('password') + #next_page = request.form.get('next_page') + + if username is not None: + user = User.get(username) + login_failed_user_id = r_cache.get('failed_login_user_id:{}'.format(username)) + # brute force by user_id + if login_failed_user_id: + login_failed_user_id = int(login_failed_user_id) + if login_failed_user_id >= 5: + error = 'Max Connection Attempts reached, Please wait {}s'.format(r_cache.ttl('failed_login_user_id:{}'.format(username))) + return render_template("login.html", error=error) + + if user and user.check_password(password): + if not check_user_role_integrity(user.get_id()): + error = 'Incorrect User ACL, Please contact your administrator' + return render_template("login.html", error=error) + login_user(user) ## TODO: use remember me ? + if user.request_password_change(): + return redirect(url_for('change_password')) + else: + return redirect(url_for('dashboard.index')) + # login failed + else: + # set brute force protection + logger.warning("Login failed, ip={}, username={}".format(current_ip, username)) + r_cache.incr('failed_login_ip:{}'.format(current_ip)) + r_cache.expire('failed_login_ip:{}'.format(current_ip), 300) + r_cache.incr('failed_login_user_id:{}'.format(username)) + r_cache.expire('failed_login_user_id:{}'.format(username), 300) + # + + error = 'Password Incorrect' + return render_template("login.html", error=error) + + return 'please provide a valid username' + + else: + #next_page = request.args.get('next') + error = request.args.get('error') + return render_template("login.html" , error=error) + +@app.route('/change_password', methods=['POST', 'GET']) +@login_required +def change_password(): + password1 = request.form.get('password1') + password2 = request.form.get('password2') + error = request.args.get('error') + + if error: + return render_template("change_password.html", error=error) + + if current_user.is_authenticated and password1!=None: + if password1==password2: + if check_password_strength(password1): + user_id = current_user.get_id() + create_user_db(user_id , password1, update=True) + return redirect(url_for('dashboard.index')) + else: + error = 'Incorrect password' + return render_template("change_password.html", error=error) + else: + error = "Passwords don't match" + return render_template("change_password.html", error=error) + else: + error = 'Please choose a new password' + return render_template("change_password.html", error=error) + +@app.route('/logout') +@login_required +def logout(): + logout_user() + return redirect(url_for('login')) + +# role error template +@app.route('/role', methods=['POST', 'GET']) +@login_required +def role(): + return render_template("error/403.html"), 403 + @app.route('/searchbox/') +@login_required +@login_analyst def searchbox(): return render_template("searchbox.html") +# ========== ERROR HANDLER ============ + +@app.errorhandler(404) +@login_required +def page_not_found(e): + # avoid endpoint enumeration + return render_template('error/404.html'), 404 # ========== INITIAL taxonomies ============ -r_serv_tags = redis.StrictRedis( - host=cfg.get("ARDB_Tags", "host"), - port=cfg.getint("ARDB_Tags", "port"), - db=cfg.getint("ARDB_Tags", "db"), - decode_responses=True) # add default ail taxonomies r_serv_tags.sadd('active_taxonomies', 'infoleak') r_serv_tags.sadd('active_taxonomies', 'gdpr') @@ -139,11 +310,6 @@ for tag in taxonomies.get('fpf').machinetags(): r_serv_tags.sadd('active_tag_fpf', tag) # ========== INITIAL tags auto export ============ -r_serv_db = redis.StrictRedis( - host=cfg.get("ARDB_DB", "host"), - port=cfg.getint("ARDB_DB", "port"), - db=cfg.getint("ARDB_DB", "db"), - decode_responses=True) infoleak_tags = taxonomies.get('infoleak').machinetags() infoleak_automatic_tags = [] for tag in taxonomies.get('infoleak').machinetags(): @@ -154,4 +320,4 @@ r_serv_db.sadd('list_export_tags', 'infoleak:submission="manual"') # ============ MAIN ============ if __name__ == "__main__": - app.run(host='0.0.0.0', port=7000, threaded=True) + app.run(host='0.0.0.0', port=7000, threaded=True, ssl_context=ssl_context) diff --git a/var/www/create_default_user.py b/var/www/create_default_user.py new file mode 100755 index 00000000..c0735611 --- /dev/null +++ b/var/www/create_default_user.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# -*-coding:UTF-8 -* + +import os +import sys +import redis +import secrets +import configparser + +sys.path.append(os.path.join(os.environ['AIL_FLASK'], 'modules')) + +from Role_Manager import create_user_db, edit_user_db, get_default_admin_token + + + +configfile = os.path.join(os.environ['AIL_BIN'], 'packages/config.cfg') +if not os.path.exists(configfile): + raise Exception('Unable to find the configuration file. \ + Did you set environment variables? \ + Or activate the virtualenv.') +cfg = configparser.ConfigParser() +cfg.read(configfile) + +r_serv = redis.StrictRedis( + host=cfg.get("ARDB_DB", "host"), + port=cfg.getint("ARDB_DB", "port"), + db=cfg.getint("ARDB_DB", "db"), + decode_responses=True) + + +if __name__ == "__main__": + + # create role_list + if not r_serv.exists('ail:all_role'): + r_serv.zadd('ail:all_role', 1, 'admin') + r_serv.zadd('ail:all_role', 2, 'analyst') + + username = 'admin@admin.test' + password = secrets.token_urlsafe() + if r_serv.exists('user_metadata:admin@admin.test'): + edit_user_db(username, password=password, role='admin') + else: + create_user_db(username, password, role='admin', default=True) + token = get_default_admin_token() + + default_passwd_file = os.path.join(os.environ['AIL_HOME'], 'DEFAULT_PASSWORD') + to_write_str = '# Password Generated by default\n# This file is deleted after the first login\n#\nemail=admin@admin.test\npassword=' + to_write_str = to_write_str + password + '\nAPI_Key=' + token + with open(default_passwd_file, 'w') as f: + f.write(to_write_str) + + print('new user created: {}'.format(username)) + print('password: {}'.format(password)) diff --git a/var/www/modules/Flask_config.py b/var/www/modules/Flask_config.py index 899a26b5..ff5ba02a 100644 --- a/var/www/modules/Flask_config.py +++ b/var/www/modules/Flask_config.py @@ -7,6 +7,7 @@ import configparser import redis import os +import re import sys # FLASK # @@ -175,6 +176,9 @@ max_dashboard_logs = int(cfg.get("Flask", "max_dashboard_logs")) crawler_enabled = cfg.getboolean("Crawler", "activate_crawler") +email_regex = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,6}' +email_regex = re.compile(email_regex) + # VT try: from virusTotalKEYS import vt_key diff --git a/var/www/modules/PasteSubmit/Flask_PasteSubmit.py b/var/www/modules/PasteSubmit/Flask_PasteSubmit.py index f8f55f83..efd0650e 100644 --- a/var/www/modules/PasteSubmit/Flask_PasteSubmit.py +++ b/var/www/modules/PasteSubmit/Flask_PasteSubmit.py @@ -7,6 +7,9 @@ import redis from flask import Flask, render_template, jsonify, request, Blueprint, url_for, redirect +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + import unicodedata import string import subprocess @@ -273,6 +276,8 @@ def hive_create_case(hive_tlp, threat_level, hive_description, hive_case_title, # ============= ROUTES ============== @PasteSubmit.route("/PasteSubmit/", methods=['GET']) +@login_required +@login_analyst def PasteSubmit_page(): #active taxonomies active_taxonomies = r_serv_tags.smembers('active_taxonomies') @@ -285,6 +290,8 @@ def PasteSubmit_page(): active_galaxies = active_galaxies) @PasteSubmit.route("/PasteSubmit/submit", methods=['POST']) +@login_required +@login_analyst def submit(): #paste_name = request.form['paste_name'] @@ -394,6 +401,8 @@ def submit(): return PasteSubmit_page() @PasteSubmit.route("/PasteSubmit/submit_status", methods=['GET']) +@login_required +@login_analyst def submit_status(): UUID = request.args.get('UUID') @@ -460,6 +469,8 @@ def submit_status(): @PasteSubmit.route("/PasteSubmit/create_misp_event", methods=['POST']) +@login_required +@login_analyst def create_misp_event(): distribution = int(request.form['misp_data[Event][distribution]']) @@ -482,6 +493,8 @@ def create_misp_event(): return 'error0' @PasteSubmit.route("/PasteSubmit/create_hive_case", methods=['POST']) +@login_required +@login_analyst def create_hive_case(): hive_tlp = int(request.form['hive_tlp']) @@ -504,6 +517,8 @@ def create_hive_case(): return 'error' @PasteSubmit.route("/PasteSubmit/edit_tag_export") +@login_required +@login_analyst def edit_tag_export(): misp_auto_events = r_serv_db.get('misp:auto-events') hive_auto_alerts = r_serv_db.get('hive:auto-alerts') @@ -568,6 +583,8 @@ def edit_tag_export(): flag_hive=flag_hive) @PasteSubmit.route("/PasteSubmit/tag_export_edited", methods=['POST']) +@login_required +@login_analyst def tag_export_edited(): tag_enabled_misp = request.form.getlist('tag_enabled_misp') tag_enabled_hive = request.form.getlist('tag_enabled_hive') @@ -592,26 +609,36 @@ def tag_export_edited(): return redirect(url_for('PasteSubmit.edit_tag_export')) @PasteSubmit.route("/PasteSubmit/enable_misp_auto_event") +@login_required +@login_analyst def enable_misp_auto_event(): r_serv_db.set('misp:auto-events', 1) return edit_tag_export() @PasteSubmit.route("/PasteSubmit/disable_misp_auto_event") +@login_required +@login_analyst def disable_misp_auto_event(): r_serv_db.set('misp:auto-events', 0) return edit_tag_export() @PasteSubmit.route("/PasteSubmit/enable_hive_auto_alert") +@login_required +@login_analyst def enable_hive_auto_alert(): r_serv_db.set('hive:auto-alerts', 1) return edit_tag_export() @PasteSubmit.route("/PasteSubmit/disable_hive_auto_alert") +@login_required +@login_analyst def disable_hive_auto_alert(): r_serv_db.set('hive:auto-alerts', 0) return edit_tag_export() @PasteSubmit.route("/PasteSubmit/add_push_tag") +@login_required +@login_analyst def add_push_tag(): tag = request.args.get('tag') if tag is not None: @@ -629,6 +656,8 @@ def add_push_tag(): return 'None args', 400 @PasteSubmit.route("/PasteSubmit/delete_push_tag") +@login_required +@login_analyst def delete_push_tag(): tag = request.args.get('tag') diff --git a/var/www/modules/Role_Manager.py b/var/www/modules/Role_Manager.py new file mode 100644 index 00000000..bb48898a --- /dev/null +++ b/var/www/modules/Role_Manager.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +# -*-coding:UTF-8 -* + +import os +import re +import redis +import bcrypt +import secrets +import configparser + +from functools import wraps +from flask_login import LoginManager, current_user, login_user, logout_user, login_required + +from flask import request, current_app + +login_manager = LoginManager() +login_manager.login_view = 'role' + +# CONFIG # +configfile = os.path.join(os.environ['AIL_BIN'], 'packages/config.cfg') +if not os.path.exists(configfile): + raise Exception('Unable to find the configuration file. \ + Did you set environment variables? \ + Or activate the virtualenv.') + +cfg = configparser.ConfigParser() +cfg.read(configfile) + +r_serv_db = redis.StrictRedis( + host=cfg.get("ARDB_DB", "host"), + port=cfg.getint("ARDB_DB", "port"), + db=cfg.getint("ARDB_DB", "db"), + decode_responses=True) + +default_passwd_file = os.path.join(os.environ['AIL_HOME'], 'DEFAULT_PASSWORD') + +regex_password = r'^(?=(.*\d){2})(?=.*[a-z])(?=.*[A-Z]).{10,100}$' +regex_password = re.compile(regex_password) + +############################################################### +############### CHECK ROLE ACCESS ################## +############################################################### + +def login_admin(func): + @wraps(func) + def decorated_view(*args, **kwargs): + if not current_user.is_authenticated: + return login_manager.unauthorized() + elif (not current_user.is_in_role('admin')): + return login_manager.unauthorized() + return func(*args, **kwargs) + return decorated_view + +def login_analyst(func): + @wraps(func) + def decorated_view(*args, **kwargs): + if not current_user.is_authenticated: + return login_manager.unauthorized() + elif (not current_user.is_in_role('analyst')): + return login_manager.unauthorized() + return func(*args, **kwargs) + return decorated_view + + + +############################################################### +############################################################### +############################################################### + +def generate_new_token(user_id): + # create user token + current_token = r_serv_db.hget('user_metadata:{}'.format(user_id), 'token') + r_serv_db.hdel('user:tokens', current_token) + token = secrets.token_urlsafe(41) + r_serv_db.hset('user:tokens', token, user_id) + r_serv_db.hset('user_metadata:{}'.format(user_id), 'token', token) + +def get_default_admin_token(): + if r_serv_db.exists('user_metadata:admin@admin.test'): + return r_serv_db.hget('user_metadata:admin@admin.test', 'token') + else: + return '' + +def create_user_db(username_id , password, default=False, role=None, update=False): + password = password.encode() + password_hash = hashing_password(password) + + # create user token + generate_new_token(username_id) + + if update: + r_serv_db.hdel('user_metadata:{}'.format(username_id), 'change_passwd') + # remove default user password file + if username_id=='admin@admin.test': + os.remove(default_passwd_file) + else: + if default: + r_serv_db.hset('user_metadata:{}'.format(username_id), 'change_passwd', True) + if role: + if role in get_all_role(): + for role_to_add in get_all_user_role(role): + r_serv_db.sadd('user_role:{}'.format(role_to_add), username_id) + r_serv_db.hset('user_metadata:{}'.format(username_id), 'role', role) + + r_serv_db.hset('user:all', username_id, password_hash) + +def edit_user_db(user_id, role, password=None): + if password: + password_hash = hashing_password(password.encode()) + r_serv_db.hset('user:all', user_id, password_hash) + + current_role = r_serv_db.hget('user_metadata:{}'.format(user_id), 'role') + if role != current_role: + request_level = get_role_level(role) + current_role = get_role_level(current_role) + + if current_role < request_level: + role_to_remove = get_user_role_by_range(current_role -1, request_level - 2) + for role_id in role_to_remove: + r_serv_db.srem('user_role:{}'.format(role_id), user_id) + r_serv_db.hset('user_metadata:{}'.format(user_id), 'role', role) + else: + role_to_add = get_user_role_by_range(request_level -1, current_role) + for role_id in role_to_add: + r_serv_db.sadd('user_role:{}'.format(role_id), user_id) + r_serv_db.hset('user_metadata:{}'.format(user_id), 'role', role) + +def delete_user_db(user_id): + if r_serv_db.exists('user_metadata:{}'.format(user_id)): + role_to_remove =get_all_role() + for role_id in role_to_remove: + r_serv_db.srem('user_role:{}'.format(role_id), user_id) + user_token = r_serv_db.hget('user_metadata:{}'.format(user_id), 'token') + r_serv_db.hdel('user:tokens', user_token) + r_serv_db.delete('user_metadata:{}'.format(user_id)) + r_serv_db.hdel('user:all', user_id) + +def hashing_password(bytes_password): + hashed = bcrypt.hashpw(bytes_password, bcrypt.gensalt()) + return hashed + +def check_password_strength(password): + result = regex_password.match(password) + if result: + return True + else: + return False + +def get_all_role(): + return r_serv_db.zrange('ail:all_role', 0, -1) + +def get_role_level(role): + return int(r_serv_db.zscore('ail:all_role', role)) + +def get_all_user_role(user_role): + current_role_val = get_role_level(user_role) + return r_serv_db.zrange('ail:all_role', current_role_val -1, -1) + +def get_all_user_upper_role(user_role): + current_role_val = get_role_level(user_role) + # remove one rank + if current_role_val > 1: + return r_serv_db.zrange('ail:all_role', 0, current_role_val -2) + else: + return [] + +def get_user_role_by_range(inf, sup): + return r_serv_db.zrange('ail:all_role', inf, sup) + +def get_user_role(user_id): + return r_serv_db.hget('user_metadata:{}'.format(user_id), 'role') + +def check_user_role_integrity(user_id): + user_role = get_user_role(user_id) + all_user_role = get_all_user_role(user_role) + res = True + for role in all_user_role: + if not r_serv_db.sismember('user_role:{}'.format(role), user_id): + res = False + upper_role = get_all_user_upper_role(user_role) + for role in upper_role: + if r_serv_db.sismember('user_role:{}'.format(role), user_id): + res = False + return res diff --git a/var/www/modules/Tags/Flask_Tags.py b/var/www/modules/Tags/Flask_Tags.py index 3cc08159..8ab81297 100644 --- a/var/www/modules/Tags/Flask_Tags.py +++ b/var/www/modules/Tags/Flask_Tags.py @@ -7,6 +7,9 @@ import redis from flask import Flask, render_template, jsonify, request, Blueprint, redirect, url_for +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + import json import datetime @@ -218,6 +221,8 @@ def update_tag_last_seen(tag, tag_first_seen, tag_last_seen): # ============= ROUTES ============== @Tags.route("/tags/", methods=['GET']) +@login_required +@login_analyst def Tags_page(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -351,6 +356,8 @@ def Tags_page(): @Tags.route("/Tags/get_all_tags") +@login_required +@login_analyst def get_all_tags(): all_tags = r_serv_tags.smembers('list_tags') @@ -373,6 +380,8 @@ def get_all_tags(): return jsonify(list_tags) @Tags.route("/Tags/get_all_tags_taxonomies") +@login_required +@login_analyst def get_all_tags_taxonomies(): taxonomies = Taxonomies() @@ -390,6 +399,8 @@ def get_all_tags_taxonomies(): return jsonify(list_tags) @Tags.route("/Tags/get_all_tags_galaxies") +@login_required +@login_analyst def get_all_tags_galaxy(): active_galaxies = r_serv_tags.smembers('active_galaxies') @@ -403,6 +414,8 @@ def get_all_tags_galaxy(): return jsonify(list_tags) @Tags.route("/Tags/get_tags_taxonomie") +@login_required +@login_analyst def get_tags_taxonomie(): taxonomie = request.args.get('taxonomie') @@ -429,6 +442,8 @@ def get_tags_taxonomie(): return 'INCORRECT INPUT' @Tags.route("/Tags/get_tags_galaxy") +@login_required +@login_analyst def get_tags_galaxy(): galaxy = request.args.get('galaxy') @@ -449,6 +464,8 @@ def get_tags_galaxy(): return 'this galaxy is disable' @Tags.route("/Tags/remove_tag") +@login_required +@login_analyst def remove_tag(): #TODO verify input @@ -460,6 +477,8 @@ def remove_tag(): return redirect(url_for('showsavedpastes.showsavedpaste', paste=path)) @Tags.route("/Tags/confirm_tag") +@login_required +@login_analyst def confirm_tag(): #TODO verify input @@ -478,6 +497,8 @@ def confirm_tag(): return 'incompatible tag' @Tags.route("/Tags/tag_validation") +@login_required +@login_analyst def tag_validation(): path = request.args.get('paste') @@ -498,6 +519,8 @@ def tag_validation(): return 'input error' @Tags.route("/Tags/addTags") +@login_required +@login_analyst def addTags(): tags = request.args.get('tags') @@ -547,6 +570,8 @@ def addTags(): @Tags.route("/Tags/taxonomies") +@login_required +@login_analyst def taxonomies(): active_taxonomies = r_serv_tags.smembers('active_taxonomies') @@ -583,6 +608,8 @@ def taxonomies(): n_tags=n_tags) @Tags.route("/Tags/edit_taxonomie") +@login_required +@login_analyst def edit_taxonomie(): taxonomies = Taxonomies() @@ -631,6 +658,8 @@ def edit_taxonomie(): return 'INVALID TAXONOMIE' @Tags.route("/Tags/disable_taxonomie") +@login_required +@login_analyst def disable_taxonomie(): taxonomies = Taxonomies() @@ -651,6 +680,8 @@ def disable_taxonomie(): @Tags.route("/Tags/active_taxonomie") +@login_required +@login_analyst def active_taxonomie(): taxonomies = Taxonomies() @@ -670,6 +701,8 @@ def active_taxonomie(): return "INCORRECT INPUT" @Tags.route("/Tags/edit_taxonomie_tag") +@login_required +@login_analyst def edit_taxonomie_tag(): taxonomies = Taxonomies() @@ -712,6 +745,8 @@ def edit_taxonomie_tag(): return "INCORRECT INPUT" @Tags.route("/Tags/galaxies") +@login_required +@login_analyst def galaxies(): active_galaxies = r_serv_tags.smembers('active_galaxies') @@ -758,6 +793,8 @@ def galaxies(): @Tags.route("/Tags/edit_galaxy") +@login_required +@login_analyst def edit_galaxy(): id = request.args.get('galaxy') @@ -825,6 +862,8 @@ def edit_galaxy(): @Tags.route("/Tags/active_galaxy") +@login_required +@login_analyst def active_galaxy(): id = request.args.get('galaxy') @@ -869,6 +908,8 @@ def active_galaxy(): @Tags.route("/Tags/disable_galaxy") +@login_required +@login_analyst def disable_galaxy(): id = request.args.get('galaxy') @@ -889,6 +930,8 @@ def disable_galaxy(): @Tags.route("/Tags/edit_galaxy_tag") +@login_required +@login_analyst def edit_galaxy_tag(): arg1 = request.args.getlist('tag_enabled') @@ -961,6 +1004,8 @@ def edit_galaxy_tag(): return "INCORRECT INPUT" @Tags.route("/Tags/tag_galaxy_info") +@login_required +@login_analyst def tag_galaxy_info(): galaxy = request.args.get('galaxy') diff --git a/var/www/modules/dashboard/Flask_dashboard.py b/var/www/modules/dashboard/Flask_dashboard.py index 7ebf6747..160d9edb 100644 --- a/var/www/modules/dashboard/Flask_dashboard.py +++ b/var/www/modules/dashboard/Flask_dashboard.py @@ -14,6 +14,9 @@ from Date import Date from flask import Flask, render_template, jsonify, request, Blueprint, url_for +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + # ============ VARIABLES ============ import Flask_config @@ -109,10 +112,14 @@ def datetime_from_utc_to_local(utc_str): # ============ ROUTES ============ @dashboard.route("/_logs") +@login_required +@login_analyst def logs(): return flask.Response(event_stream(), mimetype="text/event-stream") @dashboard.route("/_get_last_logs_json") +@login_required +@login_analyst def get_last_logs_json(): date = datetime.datetime.now().strftime("%Y%m%d") @@ -154,11 +161,15 @@ def get_last_logs_json(): @dashboard.route("/_stuff", methods=['GET']) +@login_required +@login_analyst def stuff(): return jsonify(row1=get_queues(r_serv)) @dashboard.route("/") +@login_required +@login_analyst def index(): default_minute = cfg.get("Flask", "minute_processed_paste") threshold_stucked_module = cfg.getint("Module_ModuleInformation", "threshold_stucked_module") diff --git a/var/www/modules/hashDecoded/Flask_hashDecoded.py b/var/www/modules/hashDecoded/Flask_hashDecoded.py index 1cbf4f98..a3537b1a 100644 --- a/var/www/modules/hashDecoded/Flask_hashDecoded.py +++ b/var/www/modules/hashDecoded/Flask_hashDecoded.py @@ -17,6 +17,8 @@ from hashlib import sha256 import requests from flask import Flask, render_template, jsonify, request, Blueprint, redirect, url_for, send_file +from Role_Manager import login_admin, login_analyst +from flask_login import login_required # ============ VARIABLES ============ import Flask_config @@ -474,6 +476,8 @@ def correlation_graph_node_json(correlation_type, type_id, key_id): # ============= ROUTES ============== @hashDecoded.route("/hashDecoded/all_hash_search", methods=['POST']) +@login_required +@login_analyst def all_hash_search(): date_from = request.form.get('date_from') date_to = request.form.get('date_to') @@ -483,6 +487,8 @@ def all_hash_search(): return redirect(url_for('hashDecoded.hashDecoded_page', date_from=date_from, date_to=date_to, type=type, encoding=encoding, show_decoded_files=show_decoded_files)) @hashDecoded.route("/hashDecoded/", methods=['GET']) +@login_required +@login_analyst def hashDecoded_page(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -600,6 +606,8 @@ def hashDecoded_page(): @hashDecoded.route('/hashDecoded/hash_by_type') +@login_required +@login_analyst def hash_by_type(): type = request.args.get('type') type = 'text/plain' @@ -607,12 +615,16 @@ def hash_by_type(): @hashDecoded.route('/hashDecoded/hash_hash') +@login_required +@login_analyst def hash_hash(): hash = request.args.get('hash') return render_template('hash_hash.html') @hashDecoded.route('/hashDecoded/showHash') +@login_required +@login_analyst def showHash(): hash = request.args.get('hash') #hash = 'e02055d3efaad5d656345f6a8b1b6be4fe8cb5ea' @@ -666,6 +678,8 @@ def showHash(): @hashDecoded.route('/hashDecoded/downloadHash') +@login_required +@login_analyst def downloadHash(): hash = request.args.get('hash') # sanitize hash @@ -702,6 +716,8 @@ def downloadHash(): @hashDecoded.route('/hashDecoded/hash_by_type_json') +@login_required +@login_analyst def hash_by_type_json(): type = request.args.get('type') @@ -735,6 +751,8 @@ def hash_by_type_json(): @hashDecoded.route('/hashDecoded/decoder_type_json') +@login_required +@login_analyst def decoder_type_json(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -790,6 +808,8 @@ def decoder_type_json(): @hashDecoded.route('/hashDecoded/top5_type_json') +@login_required +@login_analyst def top5_type_json(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -848,6 +868,8 @@ def top5_type_json(): @hashDecoded.route('/hashDecoded/daily_type_json') +@login_required +@login_analyst def daily_type_json(): date = request.args.get('date') @@ -867,6 +889,8 @@ def daily_type_json(): @hashDecoded.route('/hashDecoded/range_type_json') +@login_required +@login_analyst def range_type_json(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -923,6 +947,8 @@ def range_type_json(): @hashDecoded.route('/hashDecoded/hash_graph_line_json') +@login_required +@login_analyst def hash_graph_line_json(): hash = request.args.get('hash') date_from = request.args.get('date_from') @@ -952,6 +978,8 @@ def hash_graph_line_json(): @hashDecoded.route('/hashDecoded/hash_graph_node_json') +@login_required +@login_analyst def hash_graph_node_json(): hash = request.args.get('hash') @@ -1019,6 +1047,8 @@ def hash_graph_node_json(): @hashDecoded.route('/hashDecoded/hash_types') +@login_required +@login_analyst def hash_types(): date_from = 20180701 date_to = 20180706 @@ -1026,6 +1056,8 @@ def hash_types(): @hashDecoded.route('/hashDecoded/send_file_to_vt_js') +@login_required +@login_analyst def send_file_to_vt_js(): hash = request.args.get('hash') @@ -1049,6 +1081,8 @@ def send_file_to_vt_js(): @hashDecoded.route('/hashDecoded/update_vt_result') +@login_required +@login_analyst def update_vt_result(): hash = request.args.get('hash') @@ -1085,6 +1119,8 @@ def update_vt_result(): ############################ PGPDump ############################ @hashDecoded.route('/decoded/pgp_by_type_json') ## TODO: REFRACTOR +@login_required +@login_analyst def pgp_by_type_json(): type_id = request.args.get('type_id') date_from = request.args.get('date_from') @@ -1129,6 +1165,8 @@ def pgp_by_type_json(): ############################ Correlation ############################ @hashDecoded.route("/correlation/pgpdump", methods=['GET']) +@login_required +@login_analyst def pgpdump_page(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -1139,6 +1177,8 @@ def pgpdump_page(): return res @hashDecoded.route("/correlation/cryptocurrency", methods=['GET']) +@login_required +@login_analyst def cryptocurrency_page(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -1149,6 +1189,8 @@ def cryptocurrency_page(): return res @hashDecoded.route("/correlation/all_pgpdump_search", methods=['POST']) +@login_required +@login_analyst def all_pgpdump_search(): date_from = request.form.get('date_from') date_to = request.form.get('date_to') @@ -1157,6 +1199,8 @@ def all_pgpdump_search(): return redirect(url_for('hashDecoded.pgpdump_page', date_from=date_from, date_to=date_to, type_id=type_id, show_decoded_files=show_decoded_files)) @hashDecoded.route("/correlation/all_cryptocurrency_search", methods=['POST']) +@login_required +@login_analyst def all_cryptocurrency_search(): date_from = request.form.get('date_from') date_to = request.form.get('date_to') @@ -1165,6 +1209,8 @@ def all_cryptocurrency_search(): return redirect(url_for('hashDecoded.cryptocurrency_page', date_from=date_from, date_to=date_to, type_id=type_id, show_decoded_files=show_decoded_files)) @hashDecoded.route('/correlation/show_pgpdump') +@login_required +@login_analyst def show_pgpdump(): type_id = request.args.get('type_id') key_id = request.args.get('key_id') @@ -1172,36 +1218,48 @@ def show_pgpdump(): @hashDecoded.route('/correlation/show_cryptocurrency') +@login_required +@login_analyst def show_cryptocurrency(): type_id = request.args.get('type_id') key_id = request.args.get('key_id') return show_correlation('cryptocurrency', type_id, key_id) @hashDecoded.route('/correlation/cryptocurrency_range_type_json') +@login_required +@login_analyst def cryptocurrency_range_type_json(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') return correlation_type_range_type_json('cryptocurrency', date_from, date_to) @hashDecoded.route('/correlation/pgpdump_range_type_json') +@login_required +@login_analyst def pgpdump_range_type_json(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') return correlation_type_range_type_json('pgpdump', date_from, date_to) @hashDecoded.route('/correlation/pgpdump_graph_node_json') +@login_required +@login_analyst def pgpdump_graph_node_json(): type_id = request.args.get('type_id') key_id = request.args.get('key_id') return correlation_graph_node_json('pgpdump', type_id, key_id) @hashDecoded.route('/correlation/cryptocurrency_graph_node_json') +@login_required +@login_analyst def cryptocurrency_graph_node_json(): type_id = request.args.get('type_id') key_id = request.args.get('key_id') return correlation_graph_node_json('cryptocurrency', type_id, key_id) @hashDecoded.route('/correlation/pgpdump_graph_line_json') +@login_required +@login_analyst def pgpdump_graph_line_json(): type_id = request.args.get('type_id') key_id = request.args.get('key_id') @@ -1234,6 +1292,8 @@ def correlation_graph_line_json(correlation_type, type_id, key_id, date_from, da return jsonify() @hashDecoded.route('/correlation/cryptocurrency_graph_line_json') +@login_required +@login_analyst def cryptocurrency_graph_line_json(): type_id = request.args.get('type_id') key_id = request.args.get('key_id') diff --git a/var/www/modules/hiddenServices/Flask_hiddenServices.py b/var/www/modules/hiddenServices/Flask_hiddenServices.py index adc69355..12fe3177 100644 --- a/var/www/modules/hiddenServices/Flask_hiddenServices.py +++ b/var/www/modules/hiddenServices/Flask_hiddenServices.py @@ -13,6 +13,9 @@ import json from pyfaup.faup import Faup from flask import Flask, render_template, jsonify, request, send_file, Blueprint, redirect, url_for +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + from Date import Date from HiddenServices import HiddenServices @@ -239,6 +242,8 @@ def delete_auto_crawler(url): # ============= ROUTES ============== @hiddenServices.route("/crawlers/", methods=['GET']) +@login_required +@login_analyst def dashboard(): crawler_metadata_onion = get_crawler_splash_status('onion') crawler_metadata_regular = get_crawler_splash_status('regular') @@ -253,15 +258,15 @@ def dashboard(): crawler_metadata_regular=crawler_metadata_regular, statDomains_onion=statDomains_onion, statDomains_regular=statDomains_regular) -@hiddenServices.route("/hiddenServices/2", methods=['GET']) -def hiddenServices_page_test(): - return render_template("Crawler_index.html") - @hiddenServices.route("/crawlers/manual", methods=['GET']) +@login_required +@login_analyst def manual(): return render_template("Crawler_Splash_manual.html", crawler_enabled=crawler_enabled) @hiddenServices.route("/crawlers/crawler_splash_onion", methods=['GET']) +@login_required +@login_analyst def crawler_splash_onion(): type = 'onion' last_onions = get_last_domains_crawled(type) @@ -279,6 +284,8 @@ def crawler_splash_onion(): crawler_metadata=crawler_metadata, date_from=date_string, date_to=date_string) @hiddenServices.route("/crawlers/Crawler_Splash_last_by_type", methods=['GET']) +@login_required +@login_analyst def Crawler_Splash_last_by_type(): type = request.args.get('type') # verify user input @@ -302,6 +309,8 @@ def Crawler_Splash_last_by_type(): crawler_metadata=crawler_metadata, date_from=date_string, date_to=date_string) @hiddenServices.route("/crawlers/blacklisted_domains", methods=['GET']) +@login_required +@login_analyst def blacklisted_domains(): blacklist_domain = request.args.get('blacklist_domain') unblacklist_domain = request.args.get('unblacklist_domain') @@ -336,6 +345,8 @@ def blacklisted_domains(): return 'Incorrect Type' @hiddenServices.route("/crawler/blacklist_domain", methods=['GET']) +@login_required +@login_analyst def blacklist_domain(): domain = request.args.get('domain') type = request.args.get('type') @@ -357,6 +368,8 @@ def blacklist_domain(): return 'Incorrect type' @hiddenServices.route("/crawler/unblacklist_domain", methods=['GET']) +@login_required +@login_analyst def unblacklist_domain(): domain = request.args.get('domain') type = request.args.get('type') @@ -378,6 +391,8 @@ def unblacklist_domain(): return 'Incorrect type' @hiddenServices.route("/crawlers/create_spider_splash", methods=['POST']) +@login_required +@login_analyst def create_spider_splash(): url = request.form.get('url_to_crawl') automatic = request.form.get('crawler_type') @@ -464,6 +479,8 @@ def create_spider_splash(): return redirect(url_for('hiddenServices.manual')) @hiddenServices.route("/crawlers/auto_crawler", methods=['GET']) +@login_required +@login_analyst def auto_crawler(): nb_element_to_display = 100 try: @@ -516,6 +533,8 @@ def auto_crawler(): auto_crawler_domain_regular_metadata=auto_crawler_domain_regular_metadata) @hiddenServices.route("/crawlers/remove_auto_crawler", methods=['GET']) +@login_required +@login_analyst def remove_auto_crawler(): url = request.args.get('url') page = request.args.get('page') @@ -525,6 +544,8 @@ def remove_auto_crawler(): return redirect(url_for('hiddenServices.auto_crawler', page=page)) @hiddenServices.route("/crawlers/crawler_dashboard_json", methods=['GET']) +@login_required +@login_analyst def crawler_dashboard_json(): crawler_metadata_onion = get_crawler_splash_status('onion') @@ -541,6 +562,8 @@ def crawler_dashboard_json(): # # TODO: refractor @hiddenServices.route("/hiddenServices/last_crawled_domains_with_stats_json", methods=['GET']) +@login_required +@login_analyst def last_crawled_domains_with_stats_json(): last_onions = r_serv_onion.lrange('last_onion', 0 ,-1) list_onion = [] @@ -590,6 +613,8 @@ def last_crawled_domains_with_stats_json(): return jsonify({'last_onions': list_onion, 'statDomains': statDomains, 'crawler_metadata':crawler_metadata}) @hiddenServices.route("/hiddenServices/get_onions_by_daterange", methods=['POST']) +@login_required +@login_analyst def get_onions_by_daterange(): date_from = request.form.get('date_from') date_to = request.form.get('date_to') @@ -601,6 +626,8 @@ def get_onions_by_daterange(): return redirect(url_for('hiddenServices.show_domains_by_daterange', date_from=date_from, date_to=date_to, service_type=service_type, domains_up=domains_up, domains_down=domains_down, domains_tags=domains_tags)) @hiddenServices.route("/hiddenServices/show_domains_by_daterange", methods=['GET']) +@login_required +@login_analyst def show_domains_by_daterange(): date_from = request.args.get('date_from') date_to = request.args.get('date_to') @@ -705,6 +732,8 @@ def show_domains_by_daterange(): domains_tags=domains_tags, type=service_type, bootstrap_label=bootstrap_label) @hiddenServices.route("/crawlers/show_domain", methods=['GET']) +@login_required +@login_analyst def show_domain(): domain = request.args.get('domain') epoch = request.args.get('epoch') @@ -788,6 +817,8 @@ def show_domain(): domain_tags=domain_tags, screenshot=screenshot) @hiddenServices.route("/crawlers/download_domain", methods=['GET']) +@login_required +@login_analyst def download_domain(): domain = request.args.get('domain') epoch = request.args.get('epoch') @@ -839,6 +870,8 @@ def download_domain(): @hiddenServices.route("/hiddenServices/onion_son", methods=['GET']) +@login_required +@login_analyst def onion_son(): onion_domain = request.args.get('onion_domain') @@ -849,6 +882,8 @@ def onion_son(): # ============= JSON ============== @hiddenServices.route("/hiddenServices/domain_crawled_7days_json", methods=['GET']) +@login_required +@login_analyst def domain_crawled_7days_json(): type = 'onion' ## TODO: # FIXME: 404 error @@ -867,6 +902,8 @@ def domain_crawled_7days_json(): return jsonify(json_domain_stats) @hiddenServices.route('/hiddenServices/domain_crawled_by_type_json') +@login_required +@login_analyst def domain_crawled_by_type_json(): current_date = request.args.get('date') type = request.args.get('type') diff --git a/var/www/modules/rawSkeleton/Flask_rawSkeleton.py b/var/www/modules/rawSkeleton/Flask_rawSkeleton.py index d17e2b33..d767a83c 100644 --- a/var/www/modules/rawSkeleton/Flask_rawSkeleton.py +++ b/var/www/modules/rawSkeleton/Flask_rawSkeleton.py @@ -7,6 +7,9 @@ import redis from flask import Flask, render_template, jsonify, request, Blueprint +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + # ============ VARIABLES ============ import Flask_config @@ -22,6 +25,8 @@ def one(): # ============= ROUTES ============== @rawSkeleton.route("/rawSkeleton/", methods=['GET']) +@login_required +@login_analyst def skeleton_page(): return render_template("rawSkeleton.html") diff --git a/var/www/modules/restApi/Flask_restApi.py b/var/www/modules/restApi/Flask_restApi.py new file mode 100644 index 00000000..4535da28 --- /dev/null +++ b/var/www/modules/restApi/Flask_restApi.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +# -*-coding:UTF-8 -* + +''' + Flask functions and routes for the rest api +''' + +import os +import re +import sys +import json +import redis +import datetime + +from flask import Flask, render_template, jsonify, request, Blueprint, redirect, url_for, Response +from flask_login import login_required + +from functools import wraps + +# ============ VARIABLES ============ +import Flask_config + +app = Flask_config.app +cfg = Flask_config.cfg +baseUrl = Flask_config.baseUrl +r_cache = Flask_config.r_cache +r_serv_db = Flask_config.r_serv_db +r_serv_onion = Flask_config.r_serv_onion +r_serv_metadata = Flask_config.r_serv_metadata + +restApi = Blueprint('restApi', __name__, template_folder='templates') + +# ============ AUTH FUNCTIONS ============ + +def check_token_format(strg, search=re.compile(r'[^a-zA-Z0-9_-]').search): + return not bool(search(strg)) + +def verify_token(token): + if len(token) != 55: + return False + + if not check_token_format(token): + return False + + if r_serv_db.hexists('user:tokens', token): + return True + else: + return False + +# ============ DECORATOR ============ + +def token_required(funct): + @wraps(funct) + def api_token(*args, **kwargs): + data = authErrors() + if data: + return Response(json.dumps(data[0], indent=2, sort_keys=True), mimetype='application/json'), data[1] + else: + return funct(*args, **kwargs) + return api_token + +def get_auth_from_header(): + token = request.headers.get('Authorization').replace(' ', '') # remove space + return token + +def authErrors(): + # Check auth + if not request.headers.get('Authorization'): + return ({'status': 'error', 'reason': 'Authentication needed'}, 401) + token = get_auth_from_header() + data = None + # verify token format + + try: + authenticated = False + if verify_token(token): + authenticated = True + + if not authenticated: + data = ({'status': 'error', 'reason': 'Authentication failed'}, 401) + except Exception as e: + print(e) + data = ({'status': 'error', 'reason': 'Malformed Authentication String'}, 400) + if data: + return data + else: + return None + +# ============ FUNCTIONS ============ + +def one(): + return 1 + +# ============= ROUTES ============== + +# @restApi.route("/api", methods=['GET']) +# @login_required +# def api(): +# return 'api doc' + +@restApi.route("api/items", methods=['POST']) +@token_required +def items(): + item = request.args.get('id') + + return Response(json.dumps({'test': 2}), mimetype='application/json') + +# ========= REGISTRATION ========= +app.register_blueprint(restApi, url_prefix=baseUrl) diff --git a/var/www/modules/restApi/templates/api_default.html b/var/www/modules/restApi/templates/api_default.html new file mode 100644 index 00000000..cb2cc24f --- /dev/null +++ b/var/www/modules/restApi/templates/api_default.html @@ -0,0 +1,35 @@ + + + + + AIL-Framework + + + + + + + + + + + + + + + {% include 'nav_bar.html' %} + +
+
+ + {% include 'crawler/menu_sidebar.html' %} + +
+ + +
+
+
+ + + diff --git a/var/www/modules/search/Flask_search.py b/var/www/modules/search/Flask_search.py index 7405b1e9..67a518fb 100644 --- a/var/www/modules/search/Flask_search.py +++ b/var/www/modules/search/Flask_search.py @@ -11,6 +11,9 @@ import datetime import flask from flask import Flask, render_template, jsonify, request, Blueprint +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + import Paste from whoosh import index from whoosh.fields import Schema, TEXT, ID @@ -93,6 +96,8 @@ def to_iso_date(timestamp): # ============ ROUTES ============ @searches.route("/search", methods=['POST']) +@login_required +@login_analyst def search(): query = request.form['query'] q = [] @@ -180,6 +185,8 @@ def search(): @searches.route("/get_more_search_result", methods=['POST']) +@login_required +@login_analyst def get_more_search_result(): query = request.form['query'] q = [] diff --git a/var/www/modules/sentiment/Flask_sentiment.py b/var/www/modules/sentiment/Flask_sentiment.py index 9a86eaa4..af6c220c 100644 --- a/var/www/modules/sentiment/Flask_sentiment.py +++ b/var/www/modules/sentiment/Flask_sentiment.py @@ -11,6 +11,9 @@ from Date import Date import flask from flask import Flask, render_template, jsonify, request, Blueprint +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + import Paste # ============ VARIABLES ============ @@ -39,11 +42,15 @@ def get_date_range(num_day): # ============ ROUTES ============ @sentiments.route("/sentiment_analysis_trending/") +@login_required +@login_analyst def sentiment_analysis_trending(): return render_template("sentiment_analysis_trending.html") @sentiments.route("/sentiment_analysis_getplotdata/", methods=['GET']) +@login_required +@login_analyst def sentiment_analysis_getplotdata(): # Get the top providers based on number of pastes oneHour = 60*60 @@ -94,12 +101,16 @@ def sentiment_analysis_getplotdata(): @sentiments.route("/sentiment_analysis_plot_tool/") +@login_required +@login_analyst def sentiment_analysis_plot_tool(): return render_template("sentiment_analysis_plot_tool.html") @sentiments.route("/sentiment_analysis_plot_tool_getdata/", methods=['GET']) +@login_required +@login_analyst def sentiment_analysis_plot_tool_getdata(): getProviders = request.args.get('getProviders') diff --git a/var/www/modules/settings/Flask_settings.py b/var/www/modules/settings/Flask_settings.py index f8600f58..b1d89554 100644 --- a/var/www/modules/settings/Flask_settings.py +++ b/var/www/modules/settings/Flask_settings.py @@ -5,8 +5,13 @@ Flask functions and routes for the settings modules page ''' from flask import Flask, render_template, jsonify, request, Blueprint, redirect, url_for +from flask_login import login_required, current_user + +from Role_Manager import login_admin, login_analyst +from Role_Manager import create_user_db, edit_user_db, delete_user_db, check_password_strength, generate_new_token import json +import secrets import datetime import git_status @@ -22,6 +27,7 @@ max_preview_char = Flask_config.max_preview_char max_preview_modal = Flask_config.max_preview_modal REPO_ORIGIN = Flask_config.REPO_ORIGIN dict_update_description = Flask_config.dict_update_description +email_regex = Flask_config.email_regex settings = Blueprint('settings', __name__, template_folder='templates') @@ -31,8 +37,12 @@ settings = Blueprint('settings', __name__, template_folder='templates') def one(): return 1 -#def get_v1.5_update_tags_backgroud_status(): -# return '38%' +def check_email(email): + result = email_regex.match(email) + if result: + return True + else: + return False def get_git_metadata(): dict_git = {} @@ -71,20 +81,161 @@ def get_update_metadata(): dict_update['current_background_script_stat'] = r_serv_db.get('ail:current_background_script_stat') return dict_update + +def get_user_metadata(user_id): + user_metadata = {} + user_metadata['email'] = user_id + user_metadata['role'] = r_serv_db.hget('user_metadata:{}'.format(user_id), 'role') + user_metadata['api_key'] = r_serv_db.hget('user_metadata:{}'.format(user_id), 'token') + return user_metadata + +def get_users_metadata(list_users): + users = [] + for user in list_users: + users.append(get_user_metadata(user)) + return users + +def get_all_users(): + return r_serv_db.hkeys('user:all') + +def get_all_roles(): + return r_serv_db.zrange('ail:all_role', 0, -1) + # ============= ROUTES ============== @settings.route("/settings/", methods=['GET']) +@login_required +@login_analyst def settings_page(): git_metadata = get_git_metadata() current_version = r_serv_db.get('ail:version') update_metadata = get_update_metadata() + admin_level = current_user.is_in_role('admin') return render_template("settings_index.html", git_metadata=git_metadata, + admin_level=admin_level, current_version=current_version) +@settings.route("/settings/edit_profile", methods=['GET']) +@login_required +@login_analyst +def edit_profile(): + user_metadata = get_user_metadata(current_user.get_id()) + admin_level = current_user.is_in_role('admin') + return render_template("edit_profile.html", user_metadata=user_metadata, + admin_level=admin_level) + +@settings.route("/settings/new_token", methods=['GET']) +@login_required +@login_analyst +def new_token(): + generate_new_token(current_user.get_id()) + return redirect(url_for('settings.edit_profile')) + +@settings.route("/settings/new_token_user", methods=['GET']) +@login_required +@login_admin +def new_token_user(): + user_id = request.args.get('user_id') + if r_serv_db.exists('user_metadata:{}'.format(user_id)): + generate_new_token(user_id) + return redirect(url_for('settings.users_list')) + +@settings.route("/settings/create_user", methods=['GET']) +@login_required +@login_admin +def create_user(): + user_id = request.args.get('user_id') + error = request.args.get('error') + error_mail = request.args.get('error_mail') + role = None + if r_serv_db.exists('user_metadata:{}'.format(user_id)): + role = r_serv_db.hget('user_metadata:{}'.format(user_id), 'role') + else: + user_id = None + all_roles = get_all_roles() + return render_template("create_user.html", all_roles=all_roles, user_id=user_id, user_role=role, + error=error, error_mail=error_mail, + admin_level=True) + +@settings.route("/settings/create_user_post", methods=['POST']) +@login_required +@login_admin +def create_user_post(): + email = request.form.get('username') + role = request.form.get('user_role') + password1 = request.form.get('password1') + password2 = request.form.get('password2') + + all_roles = get_all_roles() + + if email and len(email)< 300 and check_email(email) and role: + if role in all_roles: + # password set + if password1 and password2: + if password1==password2: + if check_password_strength(password1): + password = password1 + else: + return render_template("create_user.html", all_roles=all_roles, error="Incorrect Password", admin_level=True) + else: + return render_template("create_user.html", all_roles=all_roles, error="Passwords don't match", admin_level=True) + # generate password + else: + password = secrets.token_urlsafe() + + if current_user.is_in_role('admin'): + # edit user + if r_serv_db.exists('user_metadata:{}'.format(email)): + if password1 and password2: + edit_user_db(email, password=password, role=role) + return redirect(url_for('settings.users_list', new_user=email, new_user_password=password, new_user_edited=True)) + else: + edit_user_db(email, role=role) + return redirect(url_for('settings.users_list', new_user=email, new_user_password='Password not changed', new_user_edited=True)) + # create user + else: + create_user_db(email, password, default=True, role=role) + return redirect(url_for('settings.users_list', new_user=email, new_user_password=password, new_user_edited=False)) + + else: + return render_template("create_user.html", all_roles=all_roles, admin_level=True) + else: + return render_template("create_user.html", all_roles=all_roles, error_mail=True, admin_level=True) + +@settings.route("/settings/users_list", methods=['GET']) +@login_required +@login_admin +def users_list(): + all_users = get_users_metadata(get_all_users()) + new_user = request.args.get('new_user') + new_user_dict = {} + if new_user: + new_user_dict['email'] = new_user + new_user_dict['edited'] = request.args.get('new_user_edited') + new_user_dict['password'] = request.args.get('new_user_password') + return render_template("users_list.html", all_users=all_users, new_user=new_user_dict, admin_level=True) + +@settings.route("/settings/edit_user", methods=['GET']) +@login_required +@login_admin +def edit_user(): + user_id = request.args.get('user_id') + return redirect(url_for('settings.create_user', user_id=user_id)) + +@settings.route("/settings/delete_user", methods=['GET']) +@login_required +@login_admin +def delete_user(): + user_id = request.args.get('user_id') + delete_user_db(user_id) + return redirect(url_for('settings.users_list')) + @settings.route("/settings/get_background_update_stats_json", methods=['GET']) +@login_required +@login_analyst def get_background_update_stats_json(): # handle :end, error update_stats = {} diff --git a/var/www/modules/settings/templates/create_user.html b/var/www/modules/settings/templates/create_user.html new file mode 100644 index 00000000..5ab809ed --- /dev/null +++ b/var/www/modules/settings/templates/create_user.html @@ -0,0 +1,153 @@ + + + + + Server Management - AIL + + + + + + + + + + + + + + + + + + {% include 'nav_bar.html' %} + +
+
+ + {% include 'settings/menu_sidebar.html' %} + +
+ + + +
+
+
+ + + + + + diff --git a/var/www/modules/settings/templates/edit_profile.html b/var/www/modules/settings/templates/edit_profile.html new file mode 100644 index 00000000..ffa65e83 --- /dev/null +++ b/var/www/modules/settings/templates/edit_profile.html @@ -0,0 +1,96 @@ + + + + + Server Management - AIL + + + + + + + + + + + + + + + + + + {% include 'nav_bar.html' %} + +
+
+ + {% include 'settings/menu_sidebar.html' %} + +
+ +
+
+
My Profile :
+
+
+ +
+
+
+
+ + + + + + + + + + + + + + + +
Email{{user_metadata['email']}}
Role{{user_metadata['role']}}
API Key + {{user_metadata['api_key']}} + +
+
+
+
+
+ +
+
+ +
+
+
+ + + + + + diff --git a/var/www/modules/settings/templates/settings_index.html b/var/www/modules/settings/templates/settings_index.html index 5b1e055a..a43afc64 100644 --- a/var/www/modules/settings/templates/settings_index.html +++ b/var/www/modules/settings/templates/settings_index.html @@ -142,7 +142,8 @@ diff --git a/var/www/modules/settings/templates/users_list.html b/var/www/modules/settings/templates/users_list.html new file mode 100644 index 00000000..0c58ab2e --- /dev/null +++ b/var/www/modules/settings/templates/users_list.html @@ -0,0 +1,122 @@ + + + + + Server Management - AIL + + + + + + + + + + + + + + + + + + + + {% include 'nav_bar.html' %} + +
+
+ + {% include 'settings/menu_sidebar.html' %} + +
+ + {% if new_user %} +
+
+
+ {% if new_user['edited']=='True' %} +
User Edited
+ {% else %} +
User Created
+ {% endif %} +
+
+

User: {{new_user['email']}}

+

Password: {{new_user['password']}}

+ Hide +
+
+
+ {% endif %} + +
+ + + + + + + + + + + {% for user in all_users %} + + + + + + + {% endfor %} + +
EmailRoleApi KeyActions
{{user['email']}}{{user['role']}} + {{user['api_key']}} + + + + + + + + +
+
+ +
+
+
+ + + + + + diff --git a/var/www/modules/showpaste/Flask_showpaste.py b/var/www/modules/showpaste/Flask_showpaste.py index b94a1073..fb990bbf 100644 --- a/var/www/modules/showpaste/Flask_showpaste.py +++ b/var/www/modules/showpaste/Flask_showpaste.py @@ -9,6 +9,10 @@ import json import os import flask from flask import Flask, render_template, jsonify, request, Blueprint, make_response, Response, send_from_directory, redirect, url_for + +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + import difflib import ssdeep @@ -380,16 +384,22 @@ def show_item_min(requested_path , content_range=0): # ============ ROUTES ============ @showsavedpastes.route("/showsavedpaste/") #completely shows the paste in a new tab +@login_required +@login_analyst def showsavedpaste(): requested_path = request.args.get('paste', '') return showpaste(0, requested_path) @showsavedpastes.route("/showsaveditem_min/") #completely shows the paste in a new tab +@login_required +@login_analyst def showsaveditem_min(): requested_path = request.args.get('paste', '') return show_item_min(requested_path) @showsavedpastes.route("/showsavedrawpaste/") #shows raw +@login_required +@login_analyst def showsavedrawpaste(): requested_path = request.args.get('paste', '') paste = Paste.Paste(requested_path) @@ -397,6 +407,8 @@ def showsavedrawpaste(): return Response(content, mimetype='text/plain') @showsavedpastes.route("/showpreviewpaste/") +@login_required +@login_analyst def showpreviewpaste(): num = request.args.get('num', '') requested_path = request.args.get('paste', '') @@ -404,6 +416,8 @@ def showpreviewpaste(): @showsavedpastes.route("/getmoredata/") +@login_required +@login_analyst def getmoredata(): requested_path = request.args.get('paste', '') paste = Paste.Paste(requested_path) @@ -412,6 +426,8 @@ def getmoredata(): return to_return @showsavedpastes.route("/showDiff/") +@login_required +@login_analyst def showDiff(): s1 = request.args.get('s1', '') s2 = request.args.get('s2', '') @@ -428,10 +444,14 @@ def showDiff(): return the_html @showsavedpastes.route('/screenshot/') +@login_required +@login_analyst def screenshot(filename): return send_from_directory(SCREENSHOT_FOLDER, filename+'.png', as_attachment=True) @showsavedpastes.route('/send_file_to_vt/', methods=['POST']) +@login_required +@login_analyst def send_file_to_vt(): b64_path = request.form['b64_path'] paste = request.form['paste'] diff --git a/var/www/modules/terms/Flask_terms.py b/var/www/modules/terms/Flask_terms.py index 1fb83bcb..f3b8c7de 100644 --- a/var/www/modules/terms/Flask_terms.py +++ b/var/www/modules/terms/Flask_terms.py @@ -11,6 +11,10 @@ import datetime import calendar import flask from flask import Flask, render_template, jsonify, request, Blueprint, url_for, redirect + +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + import re import Paste from pprint import pprint @@ -143,6 +147,8 @@ def save_tag_to_auto_push(list_tag): # ============ ROUTES ============ @terms.route("/terms_management/") +@login_required +@login_analyst def terms_management(): per_paste = request.args.get('per_paste') if per_paste == "1" or per_paste is None: @@ -261,6 +267,8 @@ def terms_management(): @terms.route("/terms_management_query_paste/") +@login_required +@login_analyst def terms_management_query_paste(): term = request.args.get('term') paste_info = [] @@ -293,6 +301,8 @@ def terms_management_query_paste(): @terms.route("/terms_management_query/") +@login_required +@login_analyst def terms_management_query(): TrackedTermsDate_Name = "TrackedTermDate" BlackListTermsDate_Name = "BlackListTermDate" @@ -315,6 +325,8 @@ def terms_management_query(): @terms.route("/terms_management_action/", methods=['GET']) +@login_required +@login_analyst def terms_management_action(): today = datetime.datetime.now() today = today.replace(microsecond=0) @@ -440,6 +452,8 @@ def terms_management_action(): return jsonify(to_return) @terms.route("/terms_management/delete_terms_tags", methods=['POST']) +@login_required +@login_analyst def delete_terms_tags(): term = request.form.get('term') tags_to_delete = request.form.getlist('tags_to_delete') @@ -452,6 +466,8 @@ def delete_terms_tags(): return 'None args', 400 @terms.route("/terms_management/delete_terms_email", methods=['GET']) +@login_required +@login_analyst def delete_terms_email(): term = request.args.get('term') email = request.args.get('email') @@ -464,6 +480,8 @@ def delete_terms_email(): @terms.route("/terms_plot_tool/") +@login_required +@login_analyst def terms_plot_tool(): term = request.args.get('term') if term is not None: @@ -473,6 +491,8 @@ def terms_plot_tool(): @terms.route("/terms_plot_tool_data/") +@login_required +@login_analyst def terms_plot_tool_data(): oneDay = 60*60*24 range_start = datetime.datetime.utcfromtimestamp(int(float(request.args.get('range_start')))) if request.args.get('range_start') is not None else 0; @@ -503,6 +523,8 @@ def terms_plot_tool_data(): @terms.route("/terms_plot_top/") +@login_required +@login_analyst def terms_plot_top(): per_paste = request.args.get('per_paste') per_paste = per_paste if per_paste is not None else 1 @@ -510,6 +532,8 @@ def terms_plot_top(): @terms.route("/terms_plot_top_data/") +@login_required +@login_analyst def terms_plot_top_data(): oneDay = 60*60*24 today = datetime.datetime.now() @@ -556,10 +580,14 @@ def terms_plot_top_data(): @terms.route("/credentials_tracker/") +@login_required +@login_analyst def credentials_tracker(): return render_template("credentials_tracker.html") @terms.route("/credentials_management_query_paste/", methods=['GET', 'POST']) +@login_required +@login_analyst def credentials_management_query_paste(): cred = request.args.get('cred') allPath = request.json['allPath'] @@ -583,6 +611,8 @@ def credentials_management_query_paste(): return jsonify(paste_info) @terms.route("/credentials_management_action/", methods=['GET']) +@login_required +@login_analyst def cred_management_action(): supplied = request.args.get('term') diff --git a/var/www/modules/trendingcharts/Flask_trendingcharts.py b/var/www/modules/trendingcharts/Flask_trendingcharts.py index ad2e5b76..a037e171 100644 --- a/var/www/modules/trendingcharts/Flask_trendingcharts.py +++ b/var/www/modules/trendingcharts/Flask_trendingcharts.py @@ -10,6 +10,9 @@ from Date import Date import flask from flask import Flask, render_template, jsonify, request, Blueprint +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + # ============ VARIABLES ============ import Flask_config @@ -36,6 +39,8 @@ def get_date_range(num_day): # ============ ROUTES ============ @trendings.route("/_progressionCharts", methods=['GET']) +@login_required +@login_analyst def progressionCharts(): attribute_name = request.args.get('attributeName') trending_name = request.args.get('trendingName') @@ -61,18 +66,24 @@ def progressionCharts(): return jsonify(keyw_value) @trendings.route("/wordstrending/") +@login_required +@login_analyst def wordstrending(): default_display = cfg.get("Flask", "default_display") return render_template("Wordstrending.html", default_display = default_display) @trendings.route("/protocolstrending/") +@login_required +@login_analyst def protocolstrending(): default_display = cfg.get("Flask", "default_display") return render_template("Protocolstrending.html", default_display = default_display) @trendings.route("/trending/") +@login_required +@login_analyst def trending(): default_display = cfg.get("Flask", "default_display") return render_template("Trending.html", default_display = default_display) diff --git a/var/www/modules/trendingmodules/Flask_trendingmodules.py b/var/www/modules/trendingmodules/Flask_trendingmodules.py index aeec0eb9..80646ecb 100644 --- a/var/www/modules/trendingmodules/Flask_trendingmodules.py +++ b/var/www/modules/trendingmodules/Flask_trendingmodules.py @@ -10,6 +10,9 @@ from Date import Date import flask from flask import Flask, render_template, jsonify, request, Blueprint +from Role_Manager import login_admin, login_analyst +from flask_login import login_required + # ============ VARIABLES ============ import Flask_config @@ -49,6 +52,8 @@ def get_date_range(num_day): # ============ ROUTES ============ @trendingmodules.route("/_moduleCharts", methods=['GET']) +@login_required +@login_analyst def modulesCharts(): keyword_name = request.args.get('keywordName') module_name = request.args.get('moduleName') @@ -75,6 +80,8 @@ def modulesCharts(): @trendingmodules.route("/_providersChart", methods=['GET']) +@login_required +@login_analyst def providersChart(): keyword_name = request.args.get('keywordName') module_name = request.args.get('moduleName') @@ -121,6 +128,8 @@ def providersChart(): @trendingmodules.route("/moduletrending/") +@login_required +@login_analyst def moduletrending(): return render_template("Moduletrending.html") diff --git a/var/www/templates/change_password.html b/var/www/templates/change_password.html new file mode 100644 index 00000000..7560d8df --- /dev/null +++ b/var/www/templates/change_password.html @@ -0,0 +1,108 @@ + + + + + + AIL-Framework + + + + + + + + + + + + + + + + + +
+ +

Change Password

+ + + + + {% if error %} +
+ {{error}} +
+ {% endif %} + + +
+
+
+
Password Requirements
+ + +
+ + + diff --git a/var/www/templates/error/403.html b/var/www/templates/error/403.html new file mode 100644 index 00000000..3193da28 --- /dev/null +++ b/var/www/templates/error/403.html @@ -0,0 +1,50 @@ + + + + + 403 - AIL + + + + + + + + + +{% include 'nav_bar.html' %} + +
+
+
+

403 Forbidden

+
+
+
+
+
+
+
+                                  ,d8       ,a8888a,     ad888888b,
+                                ,d888     ,8P"'  `"Y8,  d8"     "88
+                              ,d8" 88    ,8P        Y8,         a8P
+                            ,d8"   88    88          88      aad8"
+                          ,d8"     88    88          88      ""Y8,
+                          8888888888888  `8b        d8'         "8b
+                                   88     `8ba,  ,ad8'  Y8,     a88
+                                   88       "Y8888P"     "Y888888P'
+
+88888888888                       88           88           88           88
+88                                88           ""           88           88
+88                                88                        88           88
+88aaaaa   ,adPPYba,   8b,dPPYba,  88,dPPYba,   88   ,adPPYb,88   ,adPPYb,88   ,adPPYba,  8b,dPPYba,
+88"""""  a8"     "8a  88P'   "Y8  88P'    "8a  88  a8"    `Y88  a8"    `Y88  a8P_____88  88P'   `"8a
+88       8b       d8  88          88       d8  88  8b       88  8b       88  8PP"""""""  88       88
+88       "8a,   ,a8"  88          88b,   ,a8"  88  "8a,   ,d88  "8a,   ,d88  "8b,   ,aa  88       88
+88        `"YbbdP"'   88          8Y"Ybbd8"'   88   `"8bbdP"Y8   `"8bbdP"Y8   `"Ybbd8"'  88       88
+
+
+ + + + diff --git a/var/www/templates/error/404.html b/var/www/templates/error/404.html new file mode 100644 index 00000000..f911a550 --- /dev/null +++ b/var/www/templates/error/404.html @@ -0,0 +1,41 @@ + + + + + 404 - AIL + + + + + + + + + +{% include 'nav_bar.html' %} + +
+
+
+

404 Not Found

+
+
+
+
+
+
+
+        ,d8       ,a8888a,              ,d8
+      ,d888     ,8P"'  `"Y8,          ,d888
+    ,d8" 88    ,8P        Y8,       ,d8" 88
+  ,d8"   88    88          88     ,d8"   88
+,d8"     88    88          88   ,d8"     88
+8888888888888  `8b        d8'   8888888888888
+         88     `8ba,  ,ad8'             88
+         88       "Y8888P"               88
+
+
+ + + + diff --git a/var/www/templates/login.html b/var/www/templates/login.html new file mode 100644 index 00000000..43a3ee36 --- /dev/null +++ b/var/www/templates/login.html @@ -0,0 +1,85 @@ + + + + + + AIL-Framework + + + + + + + + + + + + + + + + + +
+ +

Please sign in

+ + + + + {% if error %} +
+ {{error}} +
+ {% endif %} + +
+ + + diff --git a/var/www/templates/nav_bar.html b/var/www/templates/nav_bar.html index 2908b984..7a813b42 100644 --- a/var/www/templates/nav_bar.html +++ b/var/www/templates/nav_bar.html @@ -33,6 +33,9 @@
  • Server Management
    • +
  • + Log Out +
    • diff --git a/var/www/templates/settings/menu_sidebar.html b/var/www/templates/settings/menu_sidebar.html index c1b58bf3..7732bee5 100644 --- a/var/www/templates/settings/menu_sidebar.html +++ b/var/www/templates/settings/menu_sidebar.html @@ -1,13 +1,58 @@
    - + +
    + My Profile +
    + + {% if admin_level %} +
    + User Management +
    + + {% endif %} + +
    diff --git a/var/www/update_thirdparty.sh b/var/www/update_thirdparty.sh index 50adf2a0..e5b66451 100755 --- a/var/www/update_thirdparty.sh +++ b/var/www/update_thirdparty.sh @@ -102,6 +102,7 @@ rm -rf temp mkdir -p ./static/image pushd static/image wget -q https://www.circl.lu/assets/images/logos/AIL.png -O AIL.png +wget -q https://www.circl.lu/assets/images/logos/AIL-logo.png -O AIL-logo.png popd if ! [[ -n "$AIL_HOME" ]]