%************************************************
\chapter{Google Keyboard (GBoard)}
\label{ch:gboard}
%************************************************

\begin{justify}
In this chapter, we briefly describe the user's and the forensics analyzer's perspective of GBoard, compare it with other popular applications, list the tested versions, present a walkthrough of the internal structure of GBoard and the rearranged data performed.
\end{justify}

\begin{justify}
GBoard is one of the most known virtual keyboard applications for mobile devices. It is closed-source, developed by Google and targets both Android and iOS devices. \parencite{GoogleGBoardBlog} Because GBoard is often found pre-installed in Android devices, most of the people use it as their primary keyboard, and it is one of the reasons why the application has more than 1 billion installations. \parencite{AndroidPoliceGBoard}
\end{justify}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.5\linewidth]{imgs/GboardScreenshot}
 \caption[Screenshot of GBoard]{Screenshot of the main view of Google Keyboard (GBoard)}
 \label{fig:GboardScreenshot}
\end{figure}

\section{Comparison with other applications}

\begin{justify}
To validate the popularity of this keyboard, a comparison was made with other popular keyboards that are available in the Google Play store\footnote{\url{https://play.google.com/store/apps/}}.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|l|}
\hline
\textbf{Name}            & \textbf{Rating (Reviews)} & \textbf{Downloads} \\ \hline
Google Keyboard (Gboard) & 4,5 (8M+)        & 1B+       \\
Microsoft SwiftKey       & 4,3 (3M+)        & 500M+     \\
GO Keyboard              & 4,5 (4M+)        & 100M+     \\
GO Keyboard Lite         & 4,4 (2M+)        & 100M+     \\
Samsung Keyboard         & 3,0 (3m+)        & 500m+     \\
\hline
\end{tabular}
\caption[Most popular keyboards in Google Play]{Comparison with the most popular keyboards in Google Play store}
\label{table:KeyboardGooglePlayComp}
\end{table}

\begin{justify}
Table \ref{table:KeyboardGooglePlayComp} shows the GBoard popularity with its high download count and user rating from Google Play store. Other relevant competitors are the Microsoft SwiftKey\footnote{\url{https://play.google.com/store/apps/details?id=com.touchtype.swiftkey}} and the Samsung Keyboard\footnote{\url{https://play.google.com/store/apps/details?id=com.samsung.keyboard.themes}}. Although they are worth researching, both are out of scope of this project. Additionally, the Samsung Keyboard could be taken into consideration even though the numbers are not relevant because Google Play only counts updated apps through their store and Samsung has its software manager to update system apps.
\end{justify}

\section{Tested versions} \label{gboard-tested-versions}

\begin{justify}
For this project, six different versions were tested in three different physical and virtual devices to determine whether a certain feature is present and check if it is feasible to analyze that data. Because GBoard bundles system libraries such as Tensorflow\footnote{\url{https://www.tensorflow.org/}}, the same version can have multiple builds for different architectures. There is also different build types for applications already installed in the device emulator and the released version in the Google Play store\footnote{\url{https://play.google.com/store}}.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Version}   & \textbf{Type}  & \textbf{Arch} & \textbf{Device} & \textbf{\acrshort{api} Level} \\ \hline
9.4.11.312687073  & preload & x86\_64   & Android Emulator & 30 \\
10.0.02.338070508 & release & x86       & Genymotion       & 29 \\
10.2.07.351353117 & release & arm64-v8a & Xiaomi MI9 SE    & 30 \\
10.4.04.361808908 & release & arm64-v8a & Xiaomi MI9 SE    & 30 \\
10.4.04.361808908 & release & x86       & Genymotion       & 29 \\
10.5.03.367007960 & release & arm64-v8a & Xiaomi MI9 SE    & 30 \\
\hline
\end{tabular}
\caption{GBoard Tested versions}
\label{table:GboardTestedVersions}
\end{table}

\begin{justify}
Table \ref{table:GboardTestedVersions} lists the tagged version associated with a type and an architecture along with the information of the device on which the application was tested on such as the device name and the Android \acrshort{api} Level.
\end{justify}

\begin{justify}
In our experiments, we did not detect differences caused by running a given version in different architectures, \gls{api} levels or build types. Nevertheless, some versions have significant differences. For instance, version 10.0.02.338070508 introduced the expression history database, while a more recent version -- 10.5.03.367007960 -- seems to no longer generate a training cache, a crucial forensic data source of user information.
\end{justify}

\section{User Perspective}

\begin{justify}
From the user perspective, GBoard is very comfortable to use due to a lot of features that make the user experience effortless. Examples include easy content sharing like \acrshort{gif}s, emojis and emoticons with smart suggestions \parencite{ramaswamy2019federated}, predictive typing that suggests the next word depending on its context \parencite{hard2019federated}, multi-language support \parencite{vanesch2019writing}, themes, personal dictionary with the users most used shortcuts and words, voice dictation, clipboard management, embedded translation and one-hand swiping. \parencite{GoogleGBoardBlog}
\end{justify}

\section{Forensics Perspective}

\begin{justify}
Naturally, these features yield digital forensic artifacts that can be valuable to forensic analyzers. Examples of such forensic artifacts are all copied text and images captured by the background service that GBoard is running, training cache data with users input information, words used frequently by the user from the personal dictionary, translation input data and much other information captured by the virtual keyboard application. Combining pieces of this data can be crucial to discover patterns or complete criminal cases.
\end{justify}

\section{Internal structure}

\begin{justify}
GBoard stores most of its internal files in the data partition of Android. Access requires root privileges, as a normal user/applications do not have enough permissions to read GBoard's data. This prevents everything other than system applications to access possible private information stored by the normal user apps.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Folder Name}   & \textbf{Description} \\ \hline
\texttt{cache} & Temporary cache files \\
\texttt{code\_cache} & - \\
\texttt{databases} & SQLite Databases \\
\texttt{files} & Normal files storage \\
\texttt{no\_backup} & - \\
\texttt{shared\_prefs} & Shared preferences \\
\hline
\end{tabular}
\caption[File structure of GBoard]{Filesystem structure of GBoard internal data folder}
\label{table:GboardInternalStructure}
\end{table}

\begin{justify}
Table \ref{table:GboardInternalStructure} shows the hierarchy of relevant directories of GBoard private storage. Some folders are unknown and empty. Next, we analyze the SQLite databases.
\end{justify}

\subsection{Databases}

\begin{justify}
GBoard internally has some SQLite3 databases to manage user data including \path{trainingcache*.db}, \path{superpacks.db}, \path{PersonalDictionary.db}, \path{gboard_clipboard.db}, \path{expression-history.db} and other databases depending on GBoard's version and on how GBoard is being used. For example, \path{expression-history.db} only appears in the most recent versions of GBoard. Tracking changes between databases was a major challenge in this project, as the release frequency of new GBoard versions is quite high. This was a hard task since internal databases do not have a proper changelog and the only way to do it is by checking one by one.
\end{justify}

\subsubsection{Training Cache}

\begin{justify}
The training cache, more specifically \path{trainingcache2.db}, is a database with user data temporarily managed in \gls{fifo} order. It holds a maximum of 1500 entries, which are used by GBoard for federating learning using TensorFlow \parencite{Hard2018Nov}. Federated learning is a deep learning scheme, where a bunch of devices contributes anonymously for the training of deep learning algorithms. According to \textcite{Hard2018Nov}, the participation of a mobile device only happens at nighttime, when the device is charging and connected to a wifi network. According to the tested versions, this database appears to be already present since version 9.4.11.312687073.
\end{justify}

\begin{justify}
Citation from \textcite{Yang2018Dec}: \say{We collect training data for this model by observing user interactions with the app: when surfacing a query suggestion to a user, a tuple (features; label) is stored in an on-device training cache, a SQLite based database with a time-to-live based data retention policy}.
\end{justify}

\begin{justify}
Forensically speaking, this database is the most important for user data collection, even temporary because it contains inserted and deleted words associated with a sequence number and a timestamp, making the data very trackable. It is important to note that some specific application fields such as password fields are not captured by this database.
\end{justify}

\begin{justify}
Moreover, interestingly, this database stores written input on incognito mode, making this functionality pretty useless in a security perspective. This is considerably bad for the user, as secret information is often written in this mode, although very crucial in a forensic perspective.
\end{justify}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.75\linewidth]{imgs/trainingcache2.pu.pdf}
 \caption[Tables from the database schema of the training cache]{Relevant tables from the database schema of the training cache}
 \label{fig:TrainingCache2Schema}
\end{figure}

\begin{justify}
Figure \ref{fig:TrainingCache2Schema} shows three tables from \path{trainingcache2.db}: \path{tf_table} holds the typed character sequences by the user, \path{d_table} stores the deleted character sequences and \path{tm_table} keeps the position in the form of a global sequence of the whitespaces and emoji characters written by the user. With the information contained in these three tables, it is possible to construct a typed phrase.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{_id} & Auto-generated primary key \\
\path{timestamp} & When the entry was created (UNIX epoch milliseconds) \\
\path{_payload} & Blob encoded in Protocol buffers \\
\hline
\end{tabular}
\caption{Common fields of training cache database tables}
\label{table:TrainingCacheCommonFieldsSchema}
\end{table}

\begin{justify}
More in-depth, each table has an auto-generated primary key \path{_id}, an associated timestamp field named \path{_timestamp} (UNIX Epoch milliseconds) and a payload named \path{_payload} encoded in a protocol buffer\footnote{\url{https://developers.google.com/protocol-buffers/}} (Table \ref{table:TrainingCacheCommonFieldsSchema}). The other fields meaning depends on the table. The table \path{sqlite_sequence} is responsible to store the last incremented values of the global sequences used at least once by the tables. This table is an internal table from SQLite.
\end{justify}

\begin{justify}
In dumps performed by \textcite{BibEntry2021Jun}, \path{_payload} field has additional data in the protocol buffer, such as smart suggestions, although, in our tested versions, none of our dumps produced this behaviour, unfortunately.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{f1} & Unknown \\
\path{f2} & Increment of the global typed sequence \\
\path{f3} & Negative accumulation of the field \path{f4} \\
\path{f4} & Character sequence size \\
\path{f5} & Representation of the deleted string \\
\path{f6} & Unknown \\
\hline
\end{tabular}
\caption{Fields description of d\_table table}
\label{table:TrainingCacheDFieldsSchema}
\end{table}

\begin{justify}
For the \path{d_table} table (Table \ref{table:TrainingCacheDFieldsSchema}), the \path{f2} field is an increment of the global typed sequence, the \path{f3} field is a negative accumulation of the field \path{f4}, which is an incrementation of the previous values by combining them with a negative operation, the \path{f4} field represents the character sequence size and the \path{f5} field is the representation of the deleted string. The fields \path{f1} and \path{f6} are unknown.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{f1} & Unknown \\
\path{f2} & Increment of the global typed sequence \\
\path{f3} & Representation of the typed string \\
\path{f4} & Unknown \\
\hline
\end{tabular}
\caption{Fields description of tf\_table table}
\label{table:TrainingCacheTFFieldsSchema}
\end{table}

\begin{justify}
For \path{tf_table} table (Table \ref{table:TrainingCacheTFFieldsSchema}), the field \path{f2} is an increment of the global typed sequence and field \path{f3} is the typed string. Fields \path{f1} and \path{f4} are unknown.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{f1} & Unknown \\
\path{f2} & Increment of the global typed sequence \\
\path{f3} & Unknown \\
\hline
\end{tabular}
\caption{Fields description of tm\_table table}
\label{table:TrainingCacheTMFieldsSchema}
\end{table}

\begin{justify}
Finally, for table \path{tm_table} (Table \ref{table:TrainingCacheTMFieldsSchema}), \path{f2} represents the global typed sequence. The fields \path{f1} and \path{f3} are unknown.
\end{justify}

\begin{justify}
Although this database contains much more tables, it was found empty with the presented test conditions, unfortunately. According to Yogesh Khatri's forensic blog \parencite{BibEntry2021Jun}, some version contains much more information when tested with a Pixel phone such as the application package name, text field path and even the type of the text field. Because we didn't have a Pixel phone for this project, we decided to not rely on this behaviour unless tested properly.
\end{justify}

\begin{justify}
Additional investigation was performed to check why those databases were not generated. By decompiling the GBoard \gls{apk} using \gls{mobsf}, we detected that, even in the most recent versions of GBoard, training cache database files names are hard written in the code and other versions of this database were also present in the strings literal table such as \path{trainingcache\%d} and \path{trainingcachev2.db}. Unfortunately, no interesting logic were found to successfully replicate the behaviour because all code was obfuscated and hard to read.
\end{justify}

\subsubsection{Clipboard}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.5\linewidth]{imgs/GboardClipboardFeature}
 \caption{GBoard clipboard management feature enabled}
 \label{fig:GBoardClipboardFeature}
\end{figure}

\begin{justify}
The clipboard database -- \path{gboard_clipboard.db} -- stores all the copied items captured by the Android clipboard. According to the Android documentation \parencite{BibEntry2020Dec}, Android provides a system service to access the \path{ClipboardManager} global instance. This service allows an app to monitor and capture all content that passes through Android clipbard. GBoard listens to this service to grab everything from the clipboard and stores an entry in the database and/or in the file storage, depending on the content type. However, note that this content is only available when the clipboard management feature is enabled in the GBoard settings, which is disabled by default.
\end{justify}

\begin{justify}
This feature has special interest from forensic analysts that are looking for credentials inside the clipboard, because, even though the keyboard detects password fields, GBoard still stores them, if copied to the clipboard. This is particularly interesting, because, as seen in Figure \ref{fig:GBoardClipboardFeature}, although the GBoard app only shows recent clipboard entries, within its databases, it keeps all data that get copied into the clipboard from the beginning.
\end{justify}

\begin{justify}
According to the tested versions, this database is already present since version \path{9.4.11.312687073}.
\end{justify}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.4\linewidth]{imgs/gboard_clipboard.pu.pdf}
 \caption{Table from the database schema of the clipboard}
 \label{fig:ClipboardSchema}
\end{figure}

\begin{justify}
Figure \ref{fig:ClipboardSchema} shows the \path{clips} table which holds all saved clipboard entries from the GBoard background \path{ClipboardManager} service listener. Table \ref{table:ClipboardSchema} lists the fields of the clips table.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{_id} & Auto-generated primary key \\
\path{text} & Representation of the copied text, if available \\
\path{html_text} & Representation in HTML of the copied text, if available \\
\path{item_type} & Partially unknown \\
\path{entity_type} & Partially unknown \\
\path{timestamp} & When the entry was created (UNIX epoch milliseconds) \\
\path{uri} & URI to the copied document \\
\path{group_id} & Unknown \\
\hline
\end{tabular}
\caption{Fields description of clips table}
\label{table:ClipboardSchema}
\end{table}

\begin{justify}
More in depth, \path{clips} table has an auto-generated primary key \path{_id}, the \path{text} field which represents the copied text, if any, the \path{html_text} field which is the \gls{html} representation of \path{text}, if it is a text entry and the \gls{html} representation is available, the \path{timestamp} field representing a timestamp in UNIX epoch formatted in milliseconds and the \path{uri} field with the \gls{uri} to the copied document, if the clipboard entry is a non-text entry, such as an image. The fields \path{item_type} and \path{entity_type} are partially unknown and \path{group_id} is completely unknown. Even with a self descriptive field name, the possible values have uncertain meaning.
\end{justify}

\subsubsection{Personal Dictionary}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.5\linewidth]{imgs/GboardDictionaryFeature}
 \caption{Personal dictionary feature of GBoard}
 \label{fig:GboardDictionaryFeature}
\end{figure}

\begin{justify}
Personal dictionary (\path{PersonalDictionary.db}) is a database that stores a user defined key-value entry of frequent words and shortcuts, respectively. This is very useful when the user writes a specific word very frequently, or in a forensics perspective, when the user writes a word but with a "masked" meaning with the objective to hide information, for example, "money" to "paper copies". According to the tested versions, this database is already present since version 9.4.11.312687073.
\end{justify}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.4\linewidth]{imgs/PersonalDictionary.pu.pdf}
 \caption{Table from the database schema of the personal dictionary}
 \label{fig:PersonalDictionarySchema}
\end{figure}

\begin{justify}
Figure \ref{fig:PersonalDictionarySchema} shows the structure of the table \path{entry} that allocates every key-value entry of the dictionary.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{_id} & Auto-generated primary key \\
\path{word} & Word to shortcut \\
\path{shortcut} & Shortcut to be used by the user \\
\path{locale} & Locale of the desired keyboard \\
\hline
\end{tabular}
\caption{Fields description of entry table}
\label{table:PersonalDictionarySchema}
\end{table}

\begin{justify}
This table has a \path{word} field to store the word to shortcut, a \path{shortcut} field that represents the shortcut and a \path{locale} field to store the locale of the desired keyboard, because GBoard supports multi-language keyboards.
\end{justify}

\subsubsection{Expression History}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.5\linewidth]{imgs/GBoardExpressionHistoryFeature}
 \caption{Expression History feature of GBoard}
 \label{fig:GBoardExpressionHistoryFeature}
\end{figure}

\begin{justify}
The expression history -- \path{expression-history.db} -- database represents the number of shares/appearances of a given emoji or emoticon. A share is considered valid simply by writing an emoji on the keyboard. From a forensic point of view, this data can be very useful to preliminarily check the type of conversation. For example, a more friendly conversation tend to have more emojis and a serious conversation much less.
\end{justify}

\begin{figure}[H]
 \centering
 \includegraphics[width=1.0\linewidth]{imgs/expression-history.pu.pdf}
 \caption{Tables from the database schema of the expression history}
 \label{fig:ExpressionHistorySchema}
\end{figure}

\begin{justify}
Figure \ref{fig:ExpressionHistorySchema} shows both \path{emoji_shares} and \path{emoticon_shares} that represents emojis and emoticons shares, respectively. Table \ref{table:ExpressionHistoryCommonFieldsSchema} details the common fields of tables \path{emoji_shares} and \path{emoticon_shares}.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|p{0.4\linewidth} | p{0.6\linewidth}|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{last_event_millis} & Last time the entry was written (UNIX epoch in milliseconds) \\
\path{shares} & Number of shares of the entry \\
\path{truncated_timestamp_millis} & Identify the month and year of the entry (UNIX epoch in milliseconds) \\
\hline
\end{tabular}
\caption{Common fields of expression history database tables}
\label{table:ExpressionHistoryCommonFieldsSchema}
\end{table}

\begin{justify}
They share \path{last_event_millis} field which represents a timestamp in UNIX epoch formatted in milliseconds and \path{shares} fields that represents the number of shares of a specific emoji/emoticon. The field \path{truncated_timestamp_millis} seems to be a truncated timestamp formatted in UNIX epoch (milliseconds) that identifies the month and year of the entry.
\end{justify}

\begin{table}[H]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Field} & \textbf{Description} \\ \hline
\path{emoji} & Text representation of the emoji \\
\path{base_variant_emoji} & Base variant of \path{emoji} \\
\path{emoticon} & Text representation of the emoticon \\
\hline
\end{tabular}
\caption{Fields description of emoji\_shares and emoticon\_shares table}
\label{table:ExpressionHistoryFieldsSchema}
\end{table}

\begin{justify}
Particularly, \path{emoji} field is a text representation of an emoji, \path{base_variant_emoji} field is a base emoji variant of \path{emoji} field, and \path{emoticon} field is a text representation of the emoticon. We hypothesize that the expression history database is linked to GBoard predictions for emojis and emoticon \cite{ramaswamy2019federated}.
\end{justify}

\subsection{Other files}

\begin{justify}
Besides the aforementioned SQLite databases, GBoard private data encompasses other files. From the forensic point, one interesting file is \path{UserHistory.dict}. The file seems to contain multiple formatted data structures in which one of them is an empty \gls{marisa} structure \parencite{MarisaTrieWebsite}. Resorting to the \path{strings} utility from \acrshort{gnu} binutils\footnote{\url{https://www.gnu.org/software/binutils/}}, we could figure out that the \path{UserHistory.dict} files holds the history of the user typed words. However, it was impractical to parse it in a consistent manner due to the complexity of the data structure.
\end{justify}

\begin{justify}
Other meaningful forensic data sources includes the \path{translate_cache} folder, as shown next.
\end{justify}

\subsubsection{Translation Cache}

\begin{figure}[H]
 \centering
 \includegraphics[width=0.5\linewidth]{imgs/GboardTranslateFeature}
 \caption{Translation feature of GBoard}
 \label{fig:GboardTranslateFeature}
\end{figure}

\begin{justify}
Translation cache is a cache produced when the embedded translator in GBoard is used. By typing words in the translator textbox, GBoard queries, through the Internet, Google translator \gls{api} and writes the translated word sequence. To speedup the process, it caches old translations to avoid repeating the query to Google Translator \gls{api}. The forensic analyzer can take this as an advantage to fetch the typed words plus the translation and metadata associated with it, namely the date/time, as it comes from Google's servers.
\end{justify}

\begin{listing}[ht]
\begin{textcode}
https://translate.google.com/translate_a/single?client=ak&dt=t&dt=ld&dt=qca&dt=rm&dt=bd&dj=1&sl=auto&tl=pt&hl=en&ie=UTF-8&oe=UTF-8&q=rice+is+good
GET
0
HTTP/1.1 200 
19
content-type: application/json; charset=utf-8
date: Fri, 19 Mar 2021 00:30:08 GMT
content-disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt
content-encoding: gzip
Cache-Control: max-age=259200, max-stale=259200
OkHttp-Sent-Millis: 1616113807248
OkHttp-Received-Millis: 1616113807374
\end{textcode}
\caption[\acrshort{http} request example to the translation \acrshort{api}]{Reduced example of an \acrshort{http} request to the translation \acrshort{api}}
\label{listing:HTTPTranslateRequest}
\end{listing}

\begin{justify}
Listing \ref{listing:HTTPTranslateRequest} presents a shortened example of a cached \gls{http} request. Although the requests are made over \gls{https}, they are stored unencrypted, except for the initial \gls{tls} handshake. Metadata like \path{date} header and all the GET parameters make context reconstruction a lot easier.
\end{justify}

\begin{listing}[ht]
\begin{jsoncode}
{
   "sentences":[
      {
         "trans":"arroz é bom",
         "orig":"rice is good",
      }
   ],
   "src":"en",
   "confidence":1.0
}
\end{jsoncode}
\caption[\acrshort{http} response example from the translation \acrshort{api}]{Reduced example of an \acrshort{http} response from the translation \acrshort{api}}
\label{listing:HTTPTranslateResponse}
\end{listing}

\begin{justify}
Listing \ref{listing:HTTPTranslateResponse} presents a reduced example of the \gls{http} response from the translation \gls{api}. With such metadata its possible to reconstruct the context with the translated text.
\end{justify}

\section{Rearranged data}

\begin{justify}
Because some data present in the databases are not directly fetchable, sometimes we needed to process it in order to get meaningful information. This way, it is possible to provide different perspectives of the information instead of just raw table data or merge information in one place.
\end{justify}

\begin{justify}
In this section, we present the \gls{sql} queries that we implemented to achieve three different perspectives of the data located in the training cache database. The goal is to obtain a high level view of the data, to yield what was effectively written on GBoard.
\end{justify}

\subsection{Training Cache}

\begin{justify}
The entries in the \textit{training cache} database is keystroke/word-based and not a continuous stream of data. Indeed, each entry is added upon a simple user task. For example, when a key is pressed a single character is registered in \path{tf_table}. The same goes when a character is deleted as it gets recorded in the \path{d_table}. Additionally when an emoji or a whitespace is inserted, an entry in \path{tm_table} is added. For suggestions or when speech-to-text is used, the entire string is registered in \path{tf_table} rather than an entry for each corresponding character.
\end{justify}

\begin{justify}
To give a meaningful high level view of the data, we reorganized its processing into three major subtables, each with its own perspective: History Timeline, Assembled Timeline and Processed History.
\end{justify}

\begin{justify}
History Timeline is a simple table ordered by time that provides the most unitary level of the information stored in the training cache with a timestamped sequence attached with a boolean to represent if a certain sequence was deleted or added.
\end{justify}

\begin{listing}[ht]
\begin{sqlcode}
SELECT time, sequence, deleted, _timestamp FROM (
	SELECT datetime(_timestamp/1000, 'unixepoch') as time, f2, f3 as sequence, false as deleted, _timestamp
	FROM tf_table
	UNION
	SELECT datetime(_timestamp/1000, 'unixepoch') as time, f2, f5 as sequence, true as deleted, _timestamp
	FROM d_table
)
ORDER BY time, f2
\end{sqlcode}
\caption{\acrshort{sql} query to fetch History Timeline data}
\label{listing:SQLHistoryTimeline}
\end{listing}

\begin{justify}
Listing \ref{listing:SQLHistoryTimeline} shows the \gls{sql} query performed in order to get the desired information. Information from the \path{f2} and \path{f3} fields of \path{tf_table} is used as inserted data and information from the \path{f2} and \path{f5} fields of \path{d_table} is used as deleted data. Then, we merge it together and order it by time and the provided accumulation sequence.
\end{justify}

\begin{justify}
Assembled Timeline is an enhanced version of the History Timeline table with the data grouped by time and deletion. This way its possible to have multiple entries together based on time and the action performed, so all deleted and inserted actions that have a common timestamp get grouped in the same single entry.
\end{justify}

\begin{listing}[ht]
\begin{sqlcode}
SELECT time, group_concat(sequence, '') as sequence, deleted, _timestamp FROM (
	SELECT datetime(_timestamp/1000, 'unixepoch') as time, f2, f3 as sequence, false as deleted, _timestamp
	FROM tf_table
	UNION
	SELECT datetime(_timestamp/1000, 'unixepoch') as time, f2, f5 as sequence, true as deleted, _timestamp
	FROM d_table
)
GROUP BY time, deleted
ORDER BY time, f2
\end{sqlcode}
\caption{\acrshort{sql} query to fetch Assembled Timeline data}
\label{listing:SQLAssembledTimeline}
\end{listing}

\begin{justify}
Listing \ref{listing:SQLAssembledTimeline} shows the \gls{sql} query performed to get the Assembled Timeline data. It is based on the query in the Listing \ref{listing:SQLHistoryTimeline}, but the information is grouped by time and the deleted boolean and then the sequence entry is concatenated.
\end{justify}

\begin{justify}
Processed History is the most refined version of the training cache information. By combining the inserted and deleted information, it is possible to "diff" the deleted characters from the inserted information, yielding an attempt to reconstruct the actual phrase written by the user. This high level representation is possibly the most useful for forensic analysts. Although some information present in this table can be inaccurate. Indeed, inaccuracies can occur when the user moves the keyboard cursor while writing. As there is no recorded data tracking cursor movements, it becomes impossible to reproduce phrases with 100\% accuracy. This problem can be circumvented by manually looking into the information present in the other generated tables and use common sense to match the words.
\end{justify}

\begin{listing}[ht]
\begin{sqlcode}
SELECT time, group_concat(sequence, '') as sequence, _timestamp
FROM (
	SELECT datetime(_timestamp/1000, 'unixepoch') as time, _timestamp, group_concat(f3, '') as sequence
	FROM tf_table
	WHERE (_timestamp, trim(f3, ' ')) NOT IN (
		SELECT _timestamp, f5
		FROM d_table
	)
	GROUP BY time, f4
	ORDER BY _timestamp, f2
)
GROUP BY _timestamp
\end{sqlcode}
\caption{\acrshort{sql} query to fetch Processed History data}
\label{listing:SQLProcessedHistory}
\end{listing}

\begin{justify}
Listing \ref{listing:SQLProcessedHistory} shows the \gls{sql} query performed to fetch the Processed History data. This query diffs the information present in \path{tf_table} and filter any associated entry in the \path{d_table}. The information is then concatenated and sorted by time, as usual.
\end{justify}

\subsection{Clipboard}

\begin{justify}
In GBoard, clipboard can have two types of information: text and images. Internally, the table \path{clips} stores every entry, but the actual content can be spread out in different places. For text, the content is stored directly in the database, but for images, it is better for them to store it directly in the file system, instead of embedding it in the database file for performance reasons.
\end{justify}

\begin{justify}
To extract the images, we need to look into the \path{uri} field in the database. If this field is not null, we can automatically know that the entry is a file. In Android, an \gls{uri} pointing to a content starts with \path{content://} followed by the package name and the service that provides that content. For example, for GBoard, an \gls{uri} pointing to a content present in the files data folder, starts with \path{content://com.google.android.inputmethod.latin.fileprovider/}. The rest is simple, just extract the remaining path, concatenate it with the root plus the \path{files} folder and read its content.
\end{justify}

\subsection{Translation Cache}

\begin{justify}
When the user uses the translation feature of the keyboard two files are stored in the cache: an \gls{http} request file and an \gls{http} response file. Each file has a corresponding hash followed by a dot and either \path{0} (zero) for the request or \path{1} (one) for the response. This way, it is possible to associate both files in a single translation query. To rearrange the date in a timeline, the \path{Date} header is extracted from the request.
\end{justify}

\begin{justify}
The response can be encoded in one of several formats. The encoding depends on the environment. The identification of the format is done through the \path{content-encoding} header field containing \path{gzip} or other values, acceptable in the \gls{http} standard \parencite{BibEntry2021Jul}.
\end{justify}

\begin{justify}
In this chapter, we presented the databases of GBoard, focusing on the most meaningful ones, from a perspective of reconstructing phrases written by the user. In the next chapter, we focus on the Autopsy digital forensic software.
\end{justify}