| Internet-Draft | hashlookup format | June 2022 |
| Dulaunoy & Huynen | Expires 25 December 2022 | [Page] |
This document describes the hashlookup output format used to express meta information of hash values seen in databases of known files. The output description includes a common semantic. The hashlookup format is used by public and private digital forensics investigations services.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 25 December 2022.¶
Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document.¶
Digital forensics is a critical field in information security and especially incident response. Providing intelligence about known set of files is crucial to avoid wasting efforts while conducting digital investigations. hashlookup format provides a common output format for diverse known databases of file hashes. Those databases are, for example, the NIST National Software Reference Library (NSRL) or Known File Filter (KFF) lists used in digital forensics software.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].¶
The hashlookup format follows the JSON [RFC8259] format. The intent of this output to be easily parsed by machines or generated by software in stream mode. Each JSON object is expressed on a single line to be processed by the client line-by-line. Examples of JSON output are presented below.¶
The main goal of the hashlookup format is to share common fields and to easily combine results from different sources. There are different reference library which are used in different uses-cases such as:¶
As there is a wide variety of sources with various granularities of information available, the hashlookup format has been made quite lax regarding the mandatory fields. The only condition is to have at least one cryptographic hash or fuzzy hashing value MUST be present in an hashlookup JSON object.¶
The following sections define the JSON fields which are permissible. The values in the key-value pairs are strings.¶
The cryptographic hashing value MUST be a JSON string. The string represents the hashed value of the file represented. The string MUST be the hexadecimal representation of the hash in upper case.¶
The fuzzy hashing value MUST be a JSON string. The string represents the hashed value of the file represented.¶
Additional fields MAY be present to describe additional metadata from the file. The value MUST be a JSON string.¶
FileName: Filename of the hashed file (NSRL)¶
FileSize: FileSize of the hashed file (NSRL)¶
CRC: CRC of the hashed file (NSRL)¶
SpecialCode: Special code of the hashed file (NSRL) as described in [NSRL-RDS]¶
OpSystemCode: OpSystemCode of the hashed file (NSRL) as described in [NSRL-RDS]¶
ProductCode: ProductCode of the hashed file (NSRL) as described in [NSRL-RDS]¶
PackageName: Package Name of the hashed file as seen in metadata of Debian pakage format, RPM or similar package managers (CIRCL)¶
PackageMaintainer: Package maintainer of the hashed file as seen in metadata of the Debian package format, RPM or similar package managers (CIRCL)¶
PackageSection: Package section of the hashed file as seen in the metadata of the Debian package format, RPM or similar package managers (CIRCL)¶
PackageVersion: Package version of the hashed file as seen in the metadata of the Debian package format, RPM or similar package managers (CIRCL)¶
KnownMalicious: List of source considering the hashed file as being malicious (CIRCL)¶
tar:gname: Group name used to create the Tar archive¶
tar:uname: User name used to create the Tar archive¶
source: Source of the hashed file¶
db: Db where the hashed file come from (if it's an import of an existing dataset)¶
insert-timestamp: When the hashed file has been inserted in the hashlooup database¶
mimetype: Guessed mimetype of the file (CIRCL)¶
nsrl-sha256: Specify if the file SHA-256 comes from the original NSRL SHA-1 to SHA-256 list¶
Two OPTIONAL fields parents and children MAY be present to represent the relationships with other hashlookup objects. The parent or children field MUST be a JSON array.
The value is a JSON string representing one the hashing algorithms. The SHA-1 is the RECOMMENDED algorithm for the relationship. Other algorithms MAY be used if SHA-1 is not available.¶
{
"CRC32": "B4DD44A4",
"FileName": "./bin/ls",
"FileSize": "110080",
"MD5": "945FEDB3A3C290D69F075F997E5320FF",
"OpSystemCode": {
"MfgCode": "1006",
"OpSystemCode": "362",
"OpSystemName": "TBD",
"OpSystemVersion": "none"
},
"ProductCode": {
"ApplicationType": "Operating System",
"Language": "English",
"MfgCode": "534",
"OpSystemCode": "599",
"ProductCode": "163568",
"ProductName": "Vinux ",
"ProductVersion": "5.1"
},
"SHA-1": "5848386F77B4C60319C68B69C4594E29959381A2",
"SHA-256": "08AC13B08BFE4407E0F0C2E12E7F5B1B5E77EB817349A5EA1D836E83CD5ACB13",
"SpecialCode": "",
"parents": [
{
"FileSize": "1090622",
"MD5": "10A2318BE86F38A6ED113E16AABAA76B",
"PackageDescription": "GNU core utilities\n This package contains the basic file, shell and text manipulation\n utilities which are expected to exist on every operating system.\n .\n Specifically, this package includes:\n arch base64 basename cat chcon chgrp chmod chown chroot cksum comm cp\n csplit cut date dd df dir dircolors dirname du echo env expand expr\n factor false flock fmt fold groups head hostid id install join link ln\n logname ls md5sum mkdir mkfifo mknod mktemp mv nice nl nohup nproc od\n paste pathchk pinky pr printenv printf ptx pwd readlink rm rmdir runcon\n sha*sum seq shred sleep sort split stat stty sum sync tac tail tee test\n timeout touch tr true truncate tsort tty uname unexpand uniq unlink\n users vdir wc who whoami yes",
"PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
"PackageName": "coreutils",
"PackageSection": "utils",
"PackageVersion": "8.21-1ubuntu5.4",
"SHA-1": "F335B669CCB7BA8A2FC8FAF315B1B4BFF9D4217F",
"SHA-256": "07995A739DAEBD60297F0E9C2B44DFAB0C735A0FE08FACC097ECE06BB4B9FA0B"
}
]
}
¶
{"FileSize": "1090622", "MD5": "10A2318BE86F38A6ED113E16AABAA76B", "PackageDescription": "GNU core utilities\n This package contains the basic file, shell and text manipulation\n utilities which are expected to exist on every operating system.\n .\n Specifically, this package includes:\n arch base64 basename cat chcon chgrp chmod chown chroot cksum comm cp\n csplit cut date dd df dir dircolors dirname du echo env expand expr\n factor false flock fmt fold groups head hostid id install join link ln\n logname ls md5sum mkdir mkfifo mknod mktemp mv nice nl nohup nproc od\n paste pathchk pinky pr printenv printf ptx pwd readlink rm rmdir runcon\n sha*sum seq shred sleep sort split stat stty sum sync tac tail tee test\n timeout touch tr true truncate tsort tty uname unexpand uniq unlink\n users vdir wc who whoami yes", "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>", "PackageName": "coreutils", "PackageSection": "utils", "PackageVersion": "8.21-1ubuntu5.4", "SHA-1": "F335B669CCB7BA8A2FC8FAF315B1B4BFF9D4217F", "SHA-256": "07995A739DAEBD60297F0E9C2B44DFAB0C735A0FE08FACC097ECE06BB4B9FA0B", "children": ["9799864E326E9DB68121471C6E72EA45152BD2C8", "A880A1F35311A7D34C9B2CA10418BD6EE94EFF58", "3E9FE88BBFE594A701522C2BAF65E209FDF7EFD9", "E03605C7911BF75BE61E54FB922B11DCF1EAAFF9", "181A05F9D249BC99FB684984C631F149CC96990C", "7EF60EF3D83B352ACC9AF996ACDC7BE3DE955705", "C687BCF2FCB74F649163AAB837C15D5800D80B31", "168CA019316332AC0A01472BDF8769801F613DCA", "6645E63EC4411B54287CB0C1321160E44B05FB87", "1989E263AF6ACF6E2869D0B4CF9433E240213C4E", "DEDFD0DA98EA7D07A5B1C7D16EACDCD0154C79B9", "C1A0868024358B0C2F0A2991BD4676B70EBF66F3", "69952AAC37372161F66DA9B07FE0BE5263A9249F", "C03D2998DE9E4332DC91586F00DAC3CA5F4761C3", "E558E7B478FAC50CCACE0E6BE28CAA19FBC74D4E", "08E3AFA387417CB0DA343837D68374A4AB8D915F", "46750D0726DB8D8E4DF870B1060E2414B64164E9", "47A2C7B8518F7E790B097785E25C8E5909AE8A39", "9D8E3FD5BE0D04815F5B0606E94C3B7C29CC8788", "FCAD50919AF0544353E950D1B52E6E2C7B98854B", "CFDE9C955B1E774EF72FB2B4A3E180EADF4CD3D0", "95C37C6826FF0C4E1C17D08285D346EE643343C0", "B35FB1B6DCEC924603F8A86677DBB0D54F5B3C18", "F8592BBAF848DEF6DC26266B15246C50D45C2095", "FF43C49690FF5E5D05CA59D7E66CEEA7E3CE00D0", "903FA7065E10EA3ED8D07598715E7AA73C38BA5A", "64D49DB459A03800399A5BB8CFFE979A604959CF", "688FCF82E39A1C44424FE9F440B69B341FC8B6E2", "0A073CD0BBFB5A33F539003F85E4D90C4CA3F2BD", "C158DC6563C36EA34408E1A36D9786A8CD95FD22", "6ECEBFE6C408316371834A52A037D91EB5BD2A1E", "2411EE623576B90A80AF6B50C47E6186F8AB6308", "2592E88C4FD1BD34207E480AEF99508090370945", "3904F363902225998E2222D67D20D01579461C15", "76FCB8813682CC8697AF1E5C6DDD5FB1DFDEA23C", "D461D21F9994EA40C68651BCA6A6D07F43A551C4", "AEE56A85B66D037EAD8C2D630194C8C46E5C061A", "69AD64E3E922A40EC3372C5DBEE57E8ABE486227", "1348C730C44DF01C0D49DB6084B5736A1D7A3BBC", "E0A97820A852729E236F7524CC23060C7BD7638E", "BF2AAAEDAB78605C43FF12450381C07C15A01D06", "071E525B285AD74B3884F1661D857E8491D2C622", "087204A60FDF211A545A5B8B900F23C0AA118333", "5848386F77B4C60319C68B69C4594E29959381A2", "E19EC3D5E7B9AEC5320CE69FAE5A61EAA7AB2143", "A57FB1F8263E8AA32B36B0D08CDA214E55DE9202", "E8FC405D941E589AA1BAB01E05AD321A049B7775", "A9DB474B1F26FAF5E01B7D545D9CA66269487053", "5E49F9779E165B4EEE965914FECBB8F7D78C59C8", "B90EAF06CF7C6F829B5D671936B59052797F78AC", "90772F507D44570ECDF615B4C232F19FC6DD9D0D", "8025FBA333B4ADCE8EFCAD284541F38C41DA09DE", "F899AA531C826504B9B494C8EA5E4532313210A0", "6AE02146CEC01123843393817D352A805227807D", "3F9E530FB7E8C37E53FFCB1ECF977E00C25F224E", "416FD291494F58795077E5EE064D92329E15DE0E", "9A9A76DA8EC26B5AA3B1F35C434466291424718F", "731D8E470227AC5AAD8432EC123984B9D052B7A0", "EBE777F05B387155604065AD4BED08D2EA3CB8B8", "787ACF5E6A560B4AEB008111F701A730AADF2EC7", "AA6F74DAD038A0DFCD4D64A002482F40BB732F10", "16C68C5BD71D93E06C2D8FC7F0F0F73954C6D7C3", "4AA2F5D8C293531D72137EAE6F71044CD46B5AD7", "99F256008B4847D716492E9763D03D049EEECE6C", "3244B00D11BA9997243A3C2D3108FC915A4BA042", "9C7C38A4B21AACB6DC51AB8A97F6CDE6704ACCB6", "781A6E4FE0CB8167CE423FC476240BDED698D676", "A71230EDAE1E5D354C672E7AB1CE92BC6059EC7C", "ACBD8B51B76889C2F55820E1C32BB26FBF67C441", "7C394AF4519CE500161DEB17857B9C057B7C74B3", "7AB1711D45DB3752B7CBE446D2E0E62D77E75EE3", "8CD8BD2875A68CD5E01F4A071B3F39F5B725B2B6", "D5FF3DB00A37075C07878A718852AFE539C7610D", "1B56B7A2684EB25235DE1708ED1371CD7879540F", "53D8A8EF35DA82BB6118BF9D8BF1ED404FC383C9", "47392375E355F56961CA42E1CFB6B98BBD484D4C", "DCBD08101D550F76DCC1B7507B152734B6F7CE7C", "F8DF08929A667BED6D7086812C319DB522E8292C", "1FB274F0883E0075D398806ACD0FA765F1118C20", "583A638A220E6FD91569F4E263771D4F89F2CC67", "39698F9FA30AE6990ACDF69280B682D90705EC32", "DE762C72E9720DB70C1897C3E0FC9C8C7D160210", "283DD2D206477E4081911F6B83950EE76503EBAD", "A7C8E7F93AE9E86A7836238644274CD73C75F5A7", "A03537A232503853D9D4C30C732CFB5F12913D93", "3F4D980B4870D5A6EB3DF334CA49C5566000C97F", "C028089A484BA0AD0166281B58DEBE0C99E5B4E7", "4A46ECF0336D55076B1C72D2459139F85DF7236C", "DF9C0BDE30B1E2C8FAF8E35CBBD3672C0AE0DFDA", "29E52E423E17C0CE93D38AA8380B0A3AF137BC7E", "BA5FD03FF246DE181DDCAA744DC2AEF4D3711762", "FE7D945A453A276E1CD005AB7A1E177D48C63A86", "DD5A833CD33617F4DCDA0220809D41FA9B1EBFC7", "4017F5B69FBB5064E51A95C856437859BA6CBA83", "427B32F83A3FC5CA0813CBACE975ECA2D6AD918B", "8F16C526AF56EF4369611718463AD4975811DD20", "FC1C867CC2D89C5985C8CB833A3980915AF7E1A1", "EB3FCFD28830CCED7C6BAA04908F574EA4F2A61F", "410D1E55EE08EF6BD7DE39DE80A02981BAF151D1", "988B560C670EDD9E2AEEE6C1D6DE584518CC57F5", "F26ABF496FC9940BB06CD4CEC3D892CF2E426B6A", "22AC7B8CA7324408A18BB95FB55AAE9E5EA85785", "F3E09D476BB01366D740EFF92453AC73C8356F43", "6DD5ACA0C43EF39FD7FFAFEEDE46986985BC84DE", "E38EDB72D805F466556C8BB796EA729CBCC04245", "D6C447B56B702869E9B429A7B47E1CE3B57BED9A", "64E23452FA980EB5DD1D4375071CB6124CF9B196", "7883EB75715603F1B89449BE617F91C65698DF38", "C328540F2D947D50EF02AC958C0DCDB51CCA0DD0", "2AC91E34FE455B026B537EA8FBA86E69E251CEE4", "672D844C60553F9B3DB9844DC29DDF49BC426F45", "EDB7ED42160C95BA2C9DD4C1E87577BD85DFE5FB", "6F4DF90B509C9392A180D7C76BB0058D4C44A4BD", "1C75F8FAF710C17B87DFD75D8390F2F2F7AF6400", "C02435F5BE0DB85C12B47E33644322631CBE7CCA", "B0C07FAC1BA571EC3054D79C40181D99CB4D01ED", "CA0B3EBCD75D8799863ADDB66E9BD378A3B88F8E", "521E4CC97D2372F821F75DB03A26938F923C002C", "CFE1C6F349F1F0611CFD3B6DD0E60EB135D7D798", "DFC1F4C951F6E09B6CFCA69127BE483279A9B5F9", "6E957B4561B081DA16886751D729170975D860BE", "A570E581D7E1D5308E88154967C3BDE3593DA50D", "99CCB36AE5BA1B6EF528C3477CDB1185744C8DA7", "A07770FC93BAA888407523BB58EE9CE97C94DB02", "56035A58036F19A2C9D312BF2730C7F970B380CA", "2C3D857765A05AF072146796B07D6A063C0B7224", "4BD3020D460E50386297CE14F9AD85B7278F9A73", "7F8BF064B328BE934582341857413CE7A387CB81", "403645FFD3A5B16F3E558947C1854FAC180C1E92", "CC125B501A779D230063BA38A7BDEB93041D82D3", "9FC7AE958DAB28FE8B6466555BDFF954BE0EC2E7", "BAE0BFB8B27FC8806194C299435AD578BC93731A", "D357E7C22254E182377A57350BB9EC870B677B32", "9C3290446A139A29D000D920E83AE5ABA264C89A", "D3582A8EF775B8E3FB4771B7B6762FBF388C131E", "DBC34938767985B8C06471483D794A1ED91529A8", "23BEFDFDBD4FDE0052EE71D6E5561F2756C85F91", "53360CB53781CACC39BED7A4484A9B8AF7D356F4", "3E98B06F33CBB14590231B74BBC277418605BE21", "C41D8DEADD83B2FFDE06CD517452A680A87A44F3", "5942742E2461BF2646FDFA48C44F1BFDE7EC37EB", "D74E21CA765F9B05DE6535799A68DCA14DE3036B", "70090B7400AC6E18BE5E1C05FB6C0EC19EBB2B5A", "AF5BE22C07A95E8D7464415A5B988D7B46F34018", "24E177F4DDB835CBA8A12F9E1372E2338A2891E9", "D805D555362790B465829C15296CC9382898FAAC", "543D2D3AA93FAC487EAD738460539FB6EC9D8D52", "B20FEC67D2246000FD86FA211DD40CAAECCEB9AD", "7A806948FE658A5BAC29A0F5CA3710533D848565", "DC949A8C7FA9FEE929930B6FB599BC82F8DC4C5A", "55DBEAFBD6A4CA81E110CF0213051C11006AC4C9", "844D6D1DB5060B26976F07C66F2F8AAD2E455F65", "E7DAF9F24A6B790F157203235278CE3F4208CB45", "B32B2AC7FFE8F658378ECA63DCA037C64A867C95", "4326E056CE3813801B5DA2848248840D2E317C94", "CEC087401C965D8AF9D4DCFBDED5AD305C86697D", "EDF9B4EABA52E2A5570D936AD74C142E3CCE1CE8", "C04C8D4891153FAF0CF8E27CBC31A556862AB783", "47073CD75ED4721EEB6FF29C52D5D871771C57A4", "BEC391C559DAE7FFC0B5D90EA7CA65028AA1D16E", "5A11AC51F28545D7965990027CA63F28C410A51A", "66F6BE00855E7A2CB618F52C99F795A055534911", "75C44CAD971780BDCE8CE499F7ED7CA235985EC5", "87AA6E1E92AA2552DC6E431E88C22774E565F14E", "21C0A42179BF4A6C2E58DDF1A1BF58C668830A50", "8AF3BAAD4074A5267E6E7D8D77D0F0B0AADFEFC4", "5B74838880C11A9FEF94FDA3964DD6BA53F812E7", "F5C5CF75A119FF8818283B9CC932217888CDC8A9", "1056EBBFBBBF2E51BCB8A48FF8038CF66095F63A", "55016AD7A449B91A5DBE59308BDD3E84261A8FF9", "F0BC5D070A82BBCD1749EBE219671FEEEAEDAED6", "3884371FFF88B50CB05D50549A1A3C106017862C", "5EEC0C5A87A28238955970C668BA4DD733A27380", "56A361D047DB9C25736638CA6C8DE5D8F5AC4A1D", "7AD82880980897DE4B9E08DFC62E7E378426F91B", "3F3AACED5AAD06D9591F9B4EDC8DA1D87CF608E4", "21E08482B1CD04B02D37FA6AC4D8B562E684E52A", "9E3B379E28C7C00AE175F23F63555EA2111D4447", "B0E6C075C30FE3F28649AF644B5BD54A4CBDC509", "B9BFA0260EA37824C42047C3EC80C83A2BEBFAC8", "DA1AE2BDBE1281B7D4E3E10D828FC370BD12572A", "3988BFCAA09F78083C23996B9D049269CF088CF2", "C2BEBF667833B55C5495753657C469AD07332007", "3E4FDE8DECFD58B5273B2B72A5D2D67804AAB27C", "3468D6009DA54AF9C6BF3E78D058D87886C9C6D2", "A6817340664ACE9688B4E9399A08024788AC03D7", "4FEC0C2432C2DF1EF03A8A63CE346179FCF3D1E4", "AB304FDC233C801E3D4129896F49A04D0F33C7E7", "B18D970EC65694033FD489FFE297ABAB9B391EC9", "8898C5F7E6CD1A1534080CB6D96003CBF0E0B5F7", "74D1EE107C5C63D4ADF5C1902F4EBD56EC1A7669", "084051009844302F09B1D5D48C2BA73F54BC8FB1", "250D7DA9FDED702FC1B3A127756367841CD851C6", "45B47C75BFE354E4DA4E6B101E4426634E56A927", "BA971742ACBB679EB80C20D0942C9F95D6BD6B7C", "861C814423F49B97077D4910FAB0C02D54EAB4B1", "44B78AF79C57BD5235883D15EC3106F96A2A5AA9", "BCF5A7EB0FD6362BB317BA69D7925002C7E1E3BE", "E0878F0243391A4537E0DF1652BE8D506FA749D1", "134A47F790EE082AB9A7E2503F01B0C164D777E8", "9352FF68CF7B5E73E7434BF138AFC2E17FB4545B", "3E6791CD56A48FABC6F637BE11D234E8068EF91F", "6F6A5B2A733ADBBB1100C44233DFBD3C5D00E4EA", "BB7E54487E24D3778C2C0EAB965AA421EA9D4D0F", "B7A599586D2BD1B69C69EB0862BC665DAAD9FC61", "2B129FA27E458F767898FD152EB65047B65238FE", "4D34641032551FBEBFFDAF751B707B90F9570C4D", "912A4B72F98C0EBEF5C235A55F49BA5EC5E463D9", "32D82E8E9873FEA2613B882461C58E13AB6BE52B", "F0C41EB583D4B17416DD70F1079E6FDEACE144FD", "014D78C0ECB1ED495D12E4FA0DA9C7A6A953945F", "4E492E947E446DF5D4B19AAB08664D65A3E604AE"]}
¶
A public hashlookup service [HASHLOOKUP-SERVICE] is provided by CIRCL and accessible as a ReST HTTP API. A software back-end implementation which produces a hashlookup format output is available [HASHLOOKUP-SERVER]. Commercial implementation such as [METALOOKUP-SERVICE] provides a compatible interface with the hashlookup format. The hashlookup project [HASHLOOKUP-IO] provides an umbrella for all the related open source projects using hashlookup format.¶
hashlookup results events might contain sensitive or confidential information. Adequate access control and encrypted transport layer shall be implemented to ensure the confidentiality of the hashlookup results.¶
hashlookup results don't imply a specific assumption concerning the maliciousness or non-maliciousness of a file. hashlookup only provides the information about the presence of a file in a specific set, known source or database.¶
The authors wish to thank all the users of the CIRCL hashlookup services for their feedback.¶