Hand drawn diagrams are essential tools for thinking and communicating in the early phases of design, yet computer based drawing tools support diagramming and sketching only poorly. Key components of computational support for early design include recognition, interpretation, and management of diagrams The paper describes the motivation for, implementation of, and initial experience with the "computer as cocktail napkin" project, a design environment based on diagrams. It explains low level recognition of glyphs, construction of higher-level recognizers, and routines for managing diagrams in the cocktail napkin prototype.