R ggplot2: add significance level to line plot

I have the following MWE in which I make some line plots with facets.

My data is in the mymelt data frame, and I want to add significance values to the plot, stored in the pvmelt data frame.

I was going to attempt this using geom_text, but I am wondering whether there is a more correct way of doing it with geom_signif.

This is the MWE I have so far:

##MWE
library(reshape2)
library(ggplot2)
set.seed(3)
mydf <- data.frame(treatment=rep(c('control','drug'),each=3), time=rep(c('week1','week2','week3'),2), fit1=runif(6, 0, 10), fit2=runif(6, 0, 10))
mymelt <- melt(mydf, measure.vars=c('fit1','fit2'))
mymelt
pvdf <- data.frame(contrast='drug - control', time=c('week1','week2','week3'), pv1=runif(3, 0, 0.075), pv2=runif(3, 0, 0.075))
pvmelt <- melt(pvdf, measure.vars=c('pv1','pv2'))
pvmelt$map.signif <- ifelse(pvmelt$value > 0.05, "", ifelse(pvmelt$value > 0.01,"*", "**"))
pvmelt
png(filename='test.png', height=500, width=800)
print(
    ggplot(mymelt, aes(x=time, y=value, group=treatment, col=treatment, shape=treatment)) +
    facet_grid(variable~.) +
    geom_line(aes(lty=treatment), size=1.5) +
    geom_point(alpha=0.5, size=4)
)
dev.off()

mymelt looks like this:

> mymelt
   treatment  time variable    value
1    control week1     fit1 1.680415
2    control week2     fit1 8.075164
3    control week3     fit1 3.849424
4       drug week1     fit1 3.277343
5       drug week2     fit1 6.021007
6       drug week3     fit1 6.043941
7    control week1     fit2 1.246334
8    control week2     fit2 2.946009
9    control week3     fit2 5.776099
10      drug week1     fit2 6.309793
11      drug week2     fit2 5.120159
12      drug week3     fit2 5.050239

While pvmelt looks like this:

> pvmelt
        contrast  time variable       value map.signif
1 drug - control week1      pv1 0.040052652          *
2 drug - control week2      pv1 0.041793708          *
3 drug - control week3      pv1 0.065093962           
4 drug - control week1      pv2 0.062228152           
5 drug - control week2      pv2 0.008358687         **
6 drug - control week3      pv2 0.052776627           

The plot I generate is this one:

But I want something like this, with pvmelt map.signif information:

What would be the best way to accomplish it?