Unfortunately the cold sores can spread to your penis. Cold Sores spread through contact so even if she touched the sore and then touched your penis it can spread. The cold sores that people generally get around their mouth is a lesser type of herpes and is usually not as painful as the other more common genital herpes.
You may not get any symptoms for some time (or at all) and if you did catch it it will most likely show when you are feeling stressed or run down. Don't be too worried though, herpes whilst not curable is treatable, and a lot more people have it than you would think. When symptoms show you can get a prescription from your doctor and things will calm down within a couple of days.
I hope everything works out well for you.