mirror of
https://github.com/cve-search/git-vuln-finder.git
synced 2024-11-22 23:07:20 +00:00
365 lines
15 KiB
Markdown
365 lines
15 KiB
Markdown
# git-vuln-finder
|
||
|
||
|
||
|
||
[](https://github.com/cve-search/git-vuln-finder/actions?query=workflow%3A%22Python+application%22)
|
||
|
||
Finding potential software vulnerabilities from git commit messages.
|
||
The output format is a JSON with the associated commit which could contain a
|
||
fix regarding a software vulnerability. The search is based on a set of regular
|
||
expressions against the commit messages only. If CVE IDs are present, those are
|
||
added automatically in the output. The input can be any git repositories or
|
||
a [GH archive source](https://www.gharchive.org/).
|
||
|
||
# Requirements
|
||
|
||
- jq (``sudo apt install jq``)
|
||
|
||
|
||
# Installation
|
||
|
||
## Use it as a library
|
||
|
||
git-vuln-finder can be install with poetry. If you don't have poetry installed, you can do the following `curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python`.
|
||
|
||
~~~bash
|
||
$ poetry install
|
||
$ poetry shell
|
||
$ git-vuln-finder -h
|
||
~~~
|
||
|
||
You can also use ``pip``. Then just import it:
|
||
|
||
~~~python
|
||
Python 3.8.0 (default, Dec 11 2019, 21:43:13)
|
||
[GCC 9.2.1 20191008] on linux
|
||
Type "help", "copyright", "credits" or "license" for more information.
|
||
>>> from git_vuln_finder import find
|
||
>>> all_potential_vulnerabilities, all_cve_found, found = find("~/git/curl")
|
||
|
||
>>> [commit for commit, summary in all_potential_vulnerabilities.items() if summary['state'] == 'cve-assigned']
|
||
['9069838b30fb3b48af0123e39f664cea683254a5', 'facb0e4662415b5f28163e853dc6742ac5fafb3d',
|
||
... snap ...
|
||
'8a75dbeb2305297640453029b7905ef51b87e8dd', '1dc43de0dccc2ea7da6dddb7b98f8d7dcf323914', '192c4f788d48f82c03e9cef40013f34370e90737', '2eb8dcf26cb37f09cffe26909a646e702dbcab66', 'fa1ae0abcde5df8d0b3283299e3f246bedf7692c', 'c11c30a8c8d727dcf5634fa0cc6ee0b4b77ddc3d', '75ca568fa1c19de4c5358fed246686de8467c238', 'a20daf90e358c1476a325ea665d533f7a27e3364', '042cc1f69ec0878f542667cb684378869f859911']
|
||
|
||
>>> print(json.dumps(all_potential_vulnerabilities['9069838b30fb3b48af0123e39f664cea683254a5'], sort_keys=True, indent=4, separators=(",", ": ")))
|
||
{
|
||
"author": "Daniel Stenberg",
|
||
"author-email": "daniel@haxx.se",
|
||
"authored_date": 1567544372,
|
||
"branches": [
|
||
"master"
|
||
],
|
||
"commit-id": "9069838b30fb3b48af0123e39f664cea683254a5",
|
||
"committed_date": 1568009674,
|
||
"cve": [
|
||
"CVE-2019-5481",
|
||
"CVE-2019-5481"
|
||
],
|
||
"language": "en",
|
||
"message": "security:read_data fix bad realloc()\n\n... that could end up a double-free\n\nCVE-2019-5481\nBug: https://curl.haxx.se/docs/CVE-2019-5481.html\n",
|
||
"origin": "https://github.com/curl/curl.git",
|
||
"origin-github-api": "https://api.github.com/repos///github.com/curl/curl/commits/9069838b30fb3b48af0123e39f664cea683254a5",
|
||
"pattern-matches": [
|
||
"double-free"
|
||
],
|
||
"pattern-selected": "(?i)(double[-| ]free|buffer overflow|double free|race[-| ]condition)",
|
||
"state": "cve-assigned",
|
||
"stats": {
|
||
"deletions": 4,
|
||
"files": 1,
|
||
"insertions": 2,
|
||
"lines": 6
|
||
},
|
||
"summary": "security:read_data fix bad realloc()",
|
||
"tags": []
|
||
}
|
||
~~~
|
||
|
||
|
||
## Use it as a command line tool
|
||
|
||
~~~bash
|
||
$ git clone https://github.com/cve-search/git-vuln-finder.git
|
||
$ cd https://github.com/cve-search/git-vuln-finder.git
|
||
$ pip install .
|
||
$ git-vuln-finder --help
|
||
~~~
|
||
|
||
You can also use pip.
|
||
``pipx`` installs scripts (system wide available) provided by Python packages
|
||
into separate virtualenvs to shield them from your system and each other.
|
||
|
||
|
||
### Usage
|
||
|
||
~~~bash
|
||
usage: git-vuln-finder [-h] [-v] [-r R] [-o O] [-s S] [-p P] [-c] [-t] [-gh GH]
|
||
|
||
Finding potential software vulnerabilities from git commit messages.
|
||
|
||
optional arguments:
|
||
-h, --help show this help message and exit
|
||
-v increase output verbosity
|
||
-r R git repository to analyse
|
||
-o O Output format: [json]
|
||
-s S State of the commit found
|
||
-p P Matching pattern to use: [vulnpatterns, cryptopatterns, cpatterns] - the pattern 'all' is used to match all the patterns at once.
|
||
-c output only a list of the CVE pattern found in commit messages (disable by default)
|
||
-t Include tags matching a specific commit
|
||
-gh GH special option for gharchive, pass a file containing a PushEvent in JSON format
|
||
|
||
More info: https://github.com/cve-search/git-vuln-finder
|
||
~~~
|
||
|
||
|
||
# Patterns
|
||
|
||
git-vuln-finder comes with 3 default patterns which can be selected to find the potential vulnerabilities described in the commit messages such as:
|
||
|
||
- [`vulnpatterns`](git_vuln_finder/patterns/en/medium/vuln) is a generic vulnerability pattern especially targeting web application and generic security commit message. Based on an academic paper.
|
||
- [`cryptopatterns`](git_vuln_finder/patterns/en/medium/crypto) is a vulnerability pattern for cryptographic errors mentioned in commit messages.
|
||
- [`cpatterns`](git_vuln_finder/patterns/en/medium/c) is a set of standard vulnerability patterns see for C/C++-like languages.
|
||
|
||
|
||
## A sample partial output from Curl git repository
|
||
|
||
~~~bash
|
||
$ git-vuln-finder -r ~/git/curl | jq .
|
||
...
|
||
"6df916d751e72fc9a1febc07bb59c4ddd886c043": {
|
||
"message": "loadlibrary: Only load system DLLs from the system directory\n\nInspiration provided by: Daniel Stenberg and Ray Satiro\n\nBug: https://curl.haxx.se/docs/adv_20160530.html\n\nRef: Windows DLL hijacking with curl, CVE-2016-4802\n",
|
||
"language": "en",
|
||
"commit-id": "6df916d751e72fc9a1febc07bb59c4ddd886c043",
|
||
"summary": "loadlibrary: Only load system DLLs from the system directory",
|
||
"stats": {
|
||
"insertions": 180,
|
||
"deletions": 8,
|
||
"lines": 188,
|
||
"files": 7
|
||
},
|
||
"author": "Steve Holme",
|
||
"author-email": "steve_holme@hotmail.com",
|
||
"authored_date": 1464555460,
|
||
"committed_date": 1464588867,
|
||
"branches": [
|
||
"master"
|
||
],
|
||
"pattern-selected": "(?i)(denial of service |\bXXE\b|remote code execution|\bopen redirect|OSVDB|\bvuln|\bCVE\b |\bXSS\b|\bReDoS\b|\bNVD\b|malicious|x−frame−options|attack|cross site |exploit|malicious|directory traversal |\bRCE\b|\bdos\b|\bXSRF \b|\bXSS\b|clickjack|session.fixation|hijack|\badvisory|\binsecure |security |\bcross−origin\b|unauthori[z|s]ed |infinite loop)",
|
||
"pattern-matches": [
|
||
"hijack"
|
||
],
|
||
"origin": "git@github.com:curl/curl.git",
|
||
"origin-github-api": "https://api.github.com/repos/curl/curl/commits/6df916d751e72fc9a1febc07bb59c4ddd886c043",
|
||
"tags": [],
|
||
"cve": [
|
||
"CVE-2016-4802"
|
||
],
|
||
"state": "cve-assigned"
|
||
},
|
||
"c2b3f264cb5210f82bdc84a3b89250a611b68dd3": {
|
||
"message": "CONNECT_ONLY: don't close connection on GSS 401/407 reponses\n\nPreviously, connections were closed immediately before the user had a\nchance to extract the socket when the proxy required Negotiate\nauthentication.\n\nThis regression was brought in with the security fix in commit\n79b9d5f1a42578f\n\nCloses #655\n",
|
||
"language": "en",
|
||
"commit-id": "c2b3f264cb5210f82bdc84a3b89250a611b68dd3",
|
||
"summary": "CONNECT_ONLY: don't close connection on GSS 401/407 reponses",
|
||
"stats": {
|
||
"insertions": 4,
|
||
"deletions": 2,
|
||
"lines": 6,
|
||
"files": 1
|
||
},
|
||
"author": "Marcel Raad",
|
||
"author-email": "raad@teamviewer.com",
|
||
"authored_date": 1455523116,
|
||
"committed_date": 1461704516,
|
||
"branches": [
|
||
"master"
|
||
],
|
||
"pattern-selected": "(?i)(denial of service |\bXXE\b|remote code execution|\bopen redirect|OSVDB|\bvuln|\bCVE\b |\bXSS\b|\bReDoS\b|\bNVD\b|malicious|x−frame−options|attack|cross site |exploit|malicious|directory traversal |\bRCE\b|\bdos\b|\bXSRF \b|\bXSS\b|clickjack|session.fixation|hijack|\badvisory|\binsecure |security |\bcross−origin\b|unauthori[z|s]ed |infinite loop)",
|
||
"pattern-matches": [
|
||
"security "
|
||
],
|
||
"origin": "git@github.com:curl/curl.git",
|
||
"origin-github-api": "https://api.github.com/repos/curl/curl/commits/c2b3f264cb5210f82bdc84a3b89250a611b68dd3",
|
||
"tags": [],
|
||
"state": "under-review"
|
||
},
|
||
...
|
||
~~~
|
||
|
||
- Extracting CVE id(s) from git messages
|
||
|
||
~~~json
|
||
"98d132cf6a879faf0147aa83ea0c07ff326260ed": {
|
||
"message": "Add a macro for testing assertion in both debug and production builds\n\nIf we have an assert then in a debug build we want an abort() to occur.\nIn a production build we wan
|
||
t the function to return an error.\n\nThis introduces a new macro to assist with that. The idea is to replace\nexisting use of OPENSSL_assert() with this new macro. The problem with\nOPENSSL
|
||
_assert() is that it aborts() on an assertion failure in both debug\nand production builds. It should never be a library's decision to abort a\nprocess (we don't get to decide when to kill t
|
||
he life support machine or\nthe nuclear reactor control system). Additionally if an attacker can\ncause a reachable assert to be hit then this can be a source of DoS attacks\ne.g. see CVE-20
|
||
17-3733, CVE-2015-0293, CVE-2011-4577 and CVE-2002-1568.\n\nReviewed-by: Tim Hudson <tjh@openssl.org>\n(Merged from https://github.com/openssl/openssl/pull/3496)",
|
||
"commit-id": "98d132cf6a879faf0147aa83ea0c07ff326260ed",
|
||
"summary": "Add a macro for testing assertion in both debug and production builds",
|
||
"stats": {
|
||
"insertions": 18,
|
||
"deletions": 0,
|
||
"lines": 18,
|
||
"files": 1
|
||
},
|
||
"author": "Matt Caswell",
|
||
"author-email": "matt@openssl.org",
|
||
"authored_date": 1495182637,
|
||
"committed_date": 1495457671,
|
||
"branches": [
|
||
"master"
|
||
],
|
||
"pattern-selected": "(?i)(denial of service |\bXXE\b|remote code execution|\bopen redirect|OSVDB|\bvuln|\bCVE\b |\bXSS\b|\bReDoS\b|\bNVD\b|malicious|x−frame−options|attack|cross site |ex
|
||
ploit|malicious|directory traversal |\bRCE\b|\bdos\b|\bXSRF \b|\bXSS\b|clickjack|session.fixation|hijack|\badvisory|\binsecure |security |\bcross−origin\b|unauthori[z|s]ed |infinite loop)",
|
||
"pattern-matches": [
|
||
"attack"
|
||
],
|
||
"cve": [
|
||
"CVE-2017-3733",
|
||
"CVE-2015-0293",
|
||
"CVE-2011-4577",
|
||
"CVE-2002-1568"
|
||
],
|
||
"state": "cve-assigned"
|
||
}
|
||
~~~
|
||
|
||
|
||
|
||
## Usage for the special gharchive option
|
||
|
||
~~~bash
|
||
$ git-vuln-finder -gh ../tests/gharchive_test.json
|
||
~~~
|
||
|
||
|
||
|
||
the value for the `gh` parameters need to be a json file, containing an array of each PushEvent you want to test.
|
||
|
||
~~~json
|
||
[
|
||
{
|
||
"id": "19351512310",
|
||
"type": "PushEvent",
|
||
"actor": {
|
||
"id": 32466128,
|
||
"login": "DavidCruciani",
|
||
"display_login": "DavidCruciani",
|
||
"gravatar_id": "",
|
||
"url": "https://api.github.com/users/DavidCruciani",
|
||
"avatar_url": "https://avatars.githubusercontent.com/u/32466128?"
|
||
},
|
||
"repo": {
|
||
"id": 424660123,
|
||
"name": "ail-project/ail-feeder-gharchive",
|
||
"url": "https://api.github.com/repos/ail-project/ail-feeder-gharchive"
|
||
},
|
||
"payload": {
|
||
"push_id": 8628652926,
|
||
"size": 1,
|
||
"distinct_size": 1,
|
||
"ref": "refs/heads/main",
|
||
"head": "910ed71a2819546a3f3bcce1ebb9e3984a8c8d86",
|
||
"before": "40a9ef5dc6b2add5184a0a58401bfe9058faa8df",
|
||
"commits": [
|
||
{
|
||
"sha": "910ed71a2819546a3f3bcce1ebb9e3984a8c8d86",
|
||
"author": {
|
||
"email": "da.cruciani@laposte.net",
|
||
"name": "David Cruciani"
|
||
},
|
||
"message": "chg: [feeder] case sensitive",
|
||
"distinct": true,
|
||
"url": "https://api.github.com/repos/ail-project/ail-feeder-gharchive/commits/910ed71a2819546a3f3bcce1ebb9e3984a8c8d86"
|
||
}
|
||
]
|
||
},
|
||
"public": true,
|
||
"created_at": "2021-12-15T16:06:43Z",
|
||
"org": {
|
||
"id": 62389074,
|
||
"login": "ail-project",
|
||
"gravatar_id": "",
|
||
"url": "https://api.github.com/orgs/ail-project",
|
||
"avatar_url": "https://avatars.githubusercontent.com/u/62389074?"
|
||
}
|
||
}
|
||
]
|
||
~~~
|
||
|
||
|
||
|
||
## Usage for import
|
||
|
||
If the goal is to import the module to use it, the method to call is `find_event`
|
||
|
||
~~~python
|
||
from git_vuln_finder import find_event
|
||
|
||
for element in event:
|
||
for i in range(0,len(element["payload"]["commits"])):
|
||
all_potential_vulnerabilities, all_cve_found, found = find_event(element["payload"]["commits"][i], element)
|
||
~~~
|
||
|
||
|
||
|
||
## Output with gharchive option
|
||
|
||
~~~json
|
||
{
|
||
"repo_name": "LeandroFChaves/gerenciador-alunos",
|
||
"message": "[UI] - Ajustes no css da aplica\u00e7\u00e3o\n\n- Adicionado padding para a exibi\u00e7\u00e3o do conte\u00fado das p\u00e1ginas;\n- Alinhado os bot\u00f5es de a\u00e7\u00f5es dos forms a direita da table",
|
||
"language": "pt",
|
||
"commit-id": "73a1c68b520853198eaac199a41d141ee96dc64d",
|
||
"author": "LeandroFChaves",
|
||
"author-email": "bbf3d4347c6affed0d9692115680849e2ace4d62@gmail.com",
|
||
"authored_date": "2021-10-01T03:00:07Z",
|
||
"branches": "refs/heads/master",
|
||
"pattern-selected": "(?i)(denial of service|\\bXXE\\b|remote code execution|\\bopen redirect|OSVDB|\\bvuln|\\bCVE\\b|\\bXSS\\b|\\bReDoS\\b|\\bNVD\\b|malicious|x\u2212frame\u2212options|attack|cross site|exploit|malicious|directory traversal|\\bRCE\\b|\\bdos\\b|\\bXSRF \\b|\\bXSS\\b|clickjack|session.fixation|hijack|\\badvisory|\\binsecure|security|\\bcross\u2212origin\\b|unauthori[z|s]ed|infinite loop)",
|
||
"pattern-matches": [
|
||
"dos"
|
||
],
|
||
"origin-github-api": "https://api.github.com/repos/LeandroFChaves/gerenciador-alunos/commits/73a1c68b520853198eaac199a41d141ee96dc64d",
|
||
"state": "under-review"
|
||
}
|
||
~~~
|
||
|
||
|
||
|
||
|
||
|
||
# Running the tests
|
||
|
||
~~~bash
|
||
$ pytest
|
||
~~~
|
||
|
||
|
||
# License and author(s)
|
||
|
||
This software is free software and licensed under the AGPL version 3.
|
||
|
||
- Copyright (c) 2019-2021 Alexandre Dulaunoy - https://github.com/adulau/
|
||
- Copyright (c) 2019-2021 All contributors to the project
|
||
|
||
# Acknowledgment
|
||
|
||
- Thanks to [Jean-Louis Huynen](https://github.com/gallypette) for the discussions about the crypto vulnerability patterns.
|
||
- Thanks to [Sebastien Tricaud](https://github.com/stricaud) for the discussions regarding native language, commit messages and external patterns.
|
||
- Thanks to [Cedric Bonhomme](https://github.com/cedricbonhomme) to make git-vuln-finder a Python library, add tests and improve the overall installation process.
|
||
- Thanks to [David Cruciani](https://github.com/DavidCruciani) for the support of [gharchive](https://www.gharchive.org/).
|
||
|
||
|
||
# Contributing
|
||
|
||
We welcome contributions for the software and especially additional vulnerability patterns. Every contributors will be added in the [AUTHORS file](./AUTHORS) and
|
||
collectively own this open source software. The contributors acknowledge the [Developer Certificate of Origin](https://developercertificate.org/).
|
||
|
||
|
||
# References
|
||
|
||
- [Notes](https://gist.github.com/adulau/dce5a6ca5c65017869bb01dfee576303#file-finding-vuln-git-commit-messages-md)
|
||
- https://csce.ucmss.com/cr/books/2017/LFS/CSREA2017/ICA2077.pdf (mainly using CVE referenced in the commit message) - archive (http://archive.is/xep9o)
|
||
- https://asankhaya.github.io/pdf/automated-identification-of-security-issues-from-commit-messages-and-bug-reports.pdf (2 main regexps)
|