Manufacturing in the Age of Multimodal Large Language Models
Abstract:
Traditional manufacturing techniques operate on a lengthy timeline, including concept development, computer-aided design (CAD) modeling, tolerances, and manufacturing constraints, followed by process planning and manufacturing instructions. A process (or manufacturing) engineer converts the CAD designs to machine-tool specifications, typically expressed in G-Code--a set of instructions used in machine-tool-controlled manufacturing processes. This pipeline has several shortcomings--inability to optimize for time, inefficient design process, not allowing extensive design exploration, and the absence of robust quality assurance practices, among many others. Lately, powerful language models (such as GPT-4) demonstrate exceptional comprehension of human-authored text and code in various scripting languages, offering a ripe opportunity for their application in manufacturing. A multimodal large language model (LLM) that can natively ingest G-Code instructions and form bidirectional mappings with natural language will significantly reduce the manual effort needed to verify, debug, index, and retrieve G-Code. In this talk, I will present an extensive evaluation of six state-of-the-art foundational large language models (LLMs) for comprehending and debugging G-code files for 3D printing. We design effective prompts to enable pre-trained LLMs to understand and manipulate G-code and test their performance on various aspects of G-code debugging and manipulation, including detection and correction of common errors and the ability to perform geometric transformations. We analyze their strengths and weaknesses for understanding complete G-code files, and the implications and limitations of using LLMs for G-code comprehension. The second part of the talk will focus on the data required to train foundational models for manufacturing. I will present SLICE-100K, a first-of-its-kind dataset of over 100,000 G-code files, along with their tessellated CAD model, LVIS (Large Vocabulary Instance Segmentation) categories, geometric properties, and renderings. We demonstrate the utility of this dataset by finetuning GPT for G-code translation from a legacy G-code format (Sailfish) to a more modern, widely used format (Marlin). Such an open large dataset will enable complex downstream tasks such as geometric reasoning, part retrieval, and shape matching; and will be the first step in developing a multimodal foundation model for digital manufacturing.
Bio:
Adarsh Krishnamurthy is an associate professor in the mechanical engineering department at Iowa State University, where he currently leads the Integrated Design and Engineering Analysis (IDEA) lab. His research interests include geometric modeling, machine learning, computer-aided design (CAD), manufacturing, GPU and parallel algorithms, biomechanics, patient-specific heart modeling, solid mechanics, and computational geometry. He is a fellow of the American Society of Mechanical Engineers (ASME) and the Plant Science Institute at Iowa State University. He received the NSF CAREER award in 2018 for developing GPU-accelerated tools for patient-specific cardiac modeling. Before joining Iowa State, he was a postdoctoral scholar at UC San Diego in the Bioengineering department. He obtained his PhD in Mechanical Engineering from UC Berkeley. His research has been funded by several federal agencies, including NSF, USDA-NIFA, ARPA-E, NASA, NIH, and ONR.