Metaheuristics are commonly used in computer science and engineering to solve optimization problems, but their potential applications in clinical trial design have remained largely unexplored. This article provides a brief overview of metaheuristics and reviews their limited use in clinical trial settings. We focus on nature-inspired metaheuristics and apply one of its exemplary algorithms, the particle swarm optimization (PSO) algorithm, to find phase I/II designs that jointly consider toxicity and efficacy. As a specific application, we demonstrate the utility of PSO in designing optimal dose-finding studies to estimate the optimal biological dose (OBD) for a continuation-ratio model with four parameters under multiple constraints. Our design improves existing designs by protecting patients from receiving doses higher than the unknown maximum tolerated dose and ensuring that the OBD is estimated with high accuracy. In addition, we show the effectiveness of metaheuristics in addressing more computationally challenging design problems by extending Simon's phase II designs to more than two stages and finding more flexible Bayesian optimal phase II designs with enhanced power.
Keywords: Continuation-ratio model; dose-finding trial; optimal biological dose; particle swarm optimization; phase I/II trial.