For 'Sweet Lou' Johnson, nothing came easy