the main script gets triggered from the WATCHDOG.pl script and is able to use the keywords on the page.
Line 6 start the phantom js
Line 11 Keywords
Line 15 Main Page crawl
Line 22 Keyword crawl
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
#!/usr/bin/perl use strict; use warnings; $|=0; system('./phantomjs-1.9.7-linux-x86_64/bin/phantomjs --webdriver='.$ARGV[0].' >> /dev/null 2>&1 &');sleep(4); print 'working '.$$.(time()-$^T).$/; my @words = ("keyword1","keyword2"); my $run =""; for(1..6){ $run ="./download.pl 'http://www.example.com/new/$_' '$ARGV[0]'"; print $run.$/; system($run." && sleep 1");sleep(10); } for (0..$#words){ my $word = $words[$_]; for(1..1){ $run ="./download.pl 'http://www.example.com/search.php?what=".$word."&page=$_' '$ARGV[0]'"; print $run.$/; system($run." && sleep 1");sleep(10); } } system("ps -e -o pid,args -dd | egrep '--webdriver=$ARGV[0]' | grep -v egrep | awk '{print $1}' | xargs kill -s 9"); print "normal exit !".$/; |
It’s an awesome paragraph for all the internet people; they will
get benefit from it I am sure.
Hello, just wanted to mention, I enjoyed this article.
It was inspiring. Keep on posting!
I really like reading through a post that may make individuals think.
Also, be grateful for permitting me to comment!