Local Storage and Session Storage in Mozilla Firefox (Part 1)

Introduction 

The Web Storage API is a mechanism that web developers can use to persist data on a user’s machine. It is often used in place of cookies when the data being stored isn’t also required to be sent to a site’s web server with each request. Web Storage is built around a simple textual key-value store, and comes in two “flavours”: Local Storage is used by a site for long-term storage – records stored indefinitely, and only cleared when the website or web application requires it (or until a user clears their browser data); and Session Storage which on the other hand is ephemeral – intended to exist for a single browser session and isolated to a single tab (i.e., session storage records stored by a site in one tab cannot be accessed by the same site in a different tab). We’ve seen all sorts of useful data stored this way, from search terms to alternative stores of browsing history to chat content, to authentication tokens and more – anything a web developer wants to persist on a user’s machine in a simple and easy to use manner.

In our coverage of Local Storage and Session Storage in Chrome we saw how these two related web technologies also shared much in common in how they were stored on a user’s computer: both LevelDB databases, containing (primarily) text data. There were of course some differences that we explored, but if you understand one, you’ll find no surprises in the other. Firefox does not treat these two storage mechanisms the same way at all, and although neither format is particularly complex, they do require a little nuance to extract the most value from each.

‍

Profiles

Like Chrome, Firefox stores its data on a per-user basis from the perspective of the operating system, but for each operating system user, there can also be multiple browser profiles. As an example, on Windows, for each operating system user you will find the main profile folders for Firefox under: “\Users\<user_goes_here>\AppData\Roaming\Mozilla\Firefox\Profiles”, with each subfolder containing a different browser profile. It is worth noting that Firefox may make a distinction between the profile information that can roam and data that should be stored only on the local machine. So, on Windows for example, an additional profile folder will be found in “\Users\<user_goes_here>\AppData\Local\Mozilla\Firefox\Profiles”; notably this second location contains the web cache, but for this article, we only need to consider the former location.

‍

Local Storage

Of the two Web Storage flavours, Local Storage operates in the way which is most in line with other artefacts related to storage APIs in Firefox. In summary: for each profile, local storage data can be found in an SQLite database per host that is using the API. This contrasts with the way that Chrome stores this data, both in the database format used, but also by segregating the data belonging to each host (Chrome stores all Local Storage data from all hosts together in a single LevelDB database). That’s the short version of the story, the full version requires a little more work. Our entry point for the Local Storage data is the “storage/default” folder in the profile folder. This folder will contain a folder for each site that has made use of one of a few different storage mechanisms. These mechanisms include Local Storage, but are not exclusive to it, as we’ll see. Just because a site is listed here, it doesn’t mean Local Storage has been used. The folders will be named based on the domain name, but with characters that are not allowed in folder names replaced with plus symbols (e.g., “http://cclsolutionsgroup.com” becomes “http+++cclsolutionsgroup”).

‍

Inside each of these website folders there will be, at least, a file named “.metadata-v2”, along with one or more folders. The folder that is related to Local Storage is named “ls”; it will only be present if Local Storage was used by the site; the other common folder is “idb” which relates to IndexedDB, which we’ll cover in a future post.

‍

‍

The metadata file isn’t strictly required for us to read the Local Storage data, but it’s only a small simple file so I’ll quickly cover its format here. The structure of this file is defined in the Mozilla source code under “dom/quota/ActorsParent.cpp” (https://hg.mozilla.org/mozilla-central/file/tip/dom/quota/ActorsParent.cpp) in the “GetDirectoryMetadata2” function. All numbers are stored big-endian, and strings are all stored as a 32-bit integer followed by that many bytes of text data.

‍

The file that we’re really interested in though, is the “data.sqlite” file which is found in the “ls” folder. Based on the file name, it is probably unsurprising to discover that this is an SQLite database. The database has two tables: “database” (which contains metadata related to the database) and “data” (which contains the local storage records themselves).

‍

The“database” table should only contain a single record, an example of which is shown below: 

• The “origin” field repeats the origin value from the “.metadata-v2” file, telling us which site this database is related to.
• “usage” gives the raw size of the data held in the database, counted in textual characters. For reasons that will become apparent, it is quite possible for this value to be larger than the size of the database on the disk.
• “last_vacuum_time” is a timestamp (encoded as microseconds since 1970-01-01) showing when the database last had an SQLite vacuum operation executed; the database is set up for incremental auto-vacuum, so it’s assumed that this represents an ejection of free-pages rather than a full rebuild of the database as would be the case with a full “VACUUM” command.
• “last_analyze_time”, one might assume, would contain a timestamp related to when the SQLite “ANALYZE” command is run, however, in all test cases it was never anything other than “0”, and the tell-tale “sqlite_stat1” table was never present in these databases. I was also unable to find any reference to this column being queried in the source code for Firefox, so I am led to believe that this field is currently unused.
• “last_vacuum_size” gives the size of the database in bytes at the time of the last vacuum command.

‍

The “data” table contains the records stored using the Local Storage API. An example of a record is shown below: 

• “key” contains the key for the local storage record as set by the website’s code
• “utf16_length” gives the length of the record’s value in UTF-16 codepoints; this may not be the actual length of the value stored in the record (because of the following two fields).
• “conversion_type” tells you how the value (which is a blob) is encoded: if this value is 0 the text will be encoded as big-endian UTF-16; if it 1, it will be encoded as UTF-8 (the meaning of these values are defined in the source code in “dom/localstorage/LSValue.h” (https://hg.mozilla.org/mozilla-central/file/tip/dom/localstorage/LSValue.h)
• “compression_type” tells you how and whether the value data is compressed: if this value is 0 then the data is uncompressed; if the value is 1 then the value data is compressed using snappy compression (the meaning of these values is also in “dom/localstorage/LSValue.h”)
• “value” contains the value of the record, stored as a blob field, stored and encoded as defined in the previous fields. 

‍

Summing up 

So, a simple database query won’t get you the full picture here, luckily our data exploration tool RabbitHole supports SQLite, snappy decompression and text decoding in the current version, so it’s still easy to view the data. For those of you who are handy with a little Python, our open source ccl_mozilla_reader contains modules for accessing data in Mozilla’s Local Storage and our fantastic, free tool Mister Skinnylegs can already extract many interesting artefacts from this data source.

In part two of this series, we will be diving into Session Storage and, along the way, learning a little extra about how Mozilla stores data related to restoring browsing sessions.

‍