Creating an AI Observer: Generative Semantic Workspaces

An experienced human Observer reading a document – such as a crime report – creates a succinct plot-like $\mathit{\text{“Working}}\mathit{\text{Memory”}}$ comprising different actors, their prototypical roles and states at any point, their evolution over time based on their interactions, and even a map of missing Semantic parts anticipating them in the future. $\mathit{\text{An}}\mathit{\text{equivalent}}\mathit{\text{AI}}\mathit{\text{Observer}}\mathit{\text{currently}}\mathit{\text{does}}\mathit{\text{not}}\mathit{\text{exist}}$. We introduce the $\mathbf{\text{[G]}}$enerative $\mathbf{\text{[S]}}$emantic $\mathbf{\text{[W]}}$orkspace (GSW) – comprising an $\mathit{\text{“Operator”}}$ and a $\mathit{\text{“Reconciler”}}$ – that leverages advancements in LLMs to create a generative-style Semantic framework, as opposed to a traditionally predefined set of lexicon labels. Given a text segment $C_n$ that describes an ongoing situation, the $\mathit{\text{Operator}}$ instantiates actor-centric Semantic maps (termed Workspace instance'' $\mathcal{W}_n$). The $\textit{Reconciler}$ resolves differences between $\mathcal{W}_n$ and a Working memory’’ ${\mathcal{M}}_n^{\ast }$ to generate the updated ${\mathcal{M}}_{n+1}^{\ast }$. GSW outperforms well-known baselines on several tasks ($\sim 94\mathrm{\%}$ vs. FST, GLEN, BertSRL - multi-sentence Semantics extraction, $\sim 15\mathrm{\%}$ vs. NLI-BERT, $\sim 35\mathrm{\%}$ vs. QA). By mirroring the real Observer, GSW provides the first step towards Spatial Computing assistants capable of understanding individual intentions and predicting future behavior.