Integrated Performance Monitoring (IPM) is an approach to applicationand architecture performance analysis that focuses on analysis of in-situ production workloads. This work presents an evaluation to date comparing conventional HPC platforms to Amazon EC2, using real applications representative of the workload at a typical supercomputing center. Architectural limits to application performance and causes significant variability are discussed.