|
Post by ezmoney on Dec 17, 2012 4:22:59 GMT -5
I do a lot of batch work and then assembled the data later. I want to automate this and get it to spawn new task as the machine load declines due to task being completed. My question is, .. is there a way to automate the process?
I am getting a lot of errors saying expired session even some very short notice ones.
Is there a way to recover from these and resume processing at some point? I have the error code trap built in but that don't seem to catch some errors and then the program is dead. Now when that happens I must restart at a point in the program or get the same data twice.. or more times for each restart.
Suggestion to Carl for your consideration. I was wondering if maybe Carl would build into his system a way to call programs and get past the logon some how. And to allow an input data file to fill in the answers rather than a keyboard like redirected input with windoes. This would greatly amplify the power of the code. Of course starting programs is nice but also there need to eventaully end or all your core is taken up, so there needs to be a way to kill the console and the program itself.
|
|
|
Post by meerkat on Dec 17, 2012 14:58:15 GMT -5
I guess I'm not exactly sure what you are trying to do. But I'll give it my best guess.
First - about timeouts. There are 2 timeouts in Run Basic, Session and Run time. - The session is the amount of time you will wait for a person to respond. For example if you have a session timeout of 2 hours, and your program ask a person a question, they have two hours to respond. - Run time is the time a program can run. This is basically to prevent loops. If you set the run time to about 30 seconds and your program continually runs for that length of time without a wait for user input it will time out.
Go the to Run Basic [Preferences] tab to set the preferences..
I assume you have a web page that your users use to get into your system. Lets assume you have a project called test. You web page probably has a link that looks something like this.
<A href="http://localhost:8008/seaside/go/runbasicpersonal?app=test" target="_blank"></A> Unless you use the RunBasic [Publish] tab and mark test as published it will ask for a login.
You can use the RUN command to run other tasks..
Not sure I answered your questions. But hope this helped. Dan
|
|
|
Post by ezmoney on Dec 28, 2012 4:13:55 GMT -5
I work from my computer to get data from the web. I have put up about 30 programs but all came back with expired session in about 1 hour. My profile is set much much higher in the 9999 run time.
I started over and some of those posted quit with expired session already and they haven't been up very long. The rest are still running... Mmmm something strange is going on here and I don't know what it is or how to solve it.. Can anybody clue me in?
|
|
|
Post by StefanPendl on Dec 28, 2012 10:08:10 GMT -5
The preferences tab is described in the quick start guide.
The runtime limit is the amount of time a program can run without user intervention. The session limit is the amount of time a program can run at all.
It seems that your session limit is still at 1 hour (= 3600 seconds), so you have to increase that too.
|
|
|
Post by meerkat on Dec 28, 2012 10:55:09 GMT -5
It's hard to know what you are trying to do without seeing some code.
However you could set a timeout in your web page and have it run itself when it times out.
This is set to timeout in about 28 Hours.
<script type="text/JavaScript"> <!-- setTimeout("location.href = 'http://localhost:8008/seaside/go/runbasicpersonal?app=yourApp';",99999); --> </script>
|
|
|
Post by ezmoney on Jan 3, 2013 3:21:28 GMT -5
Basically I do the open #1, git input file Open #4 for output limit=1000 [loop] read the git$
ss$=httpget$(git$) data$=function(ss$)
store the result
print #4,data$ rec=rec+1 if rec < limit then [again] close #4 close #1 end
I am getting a lot of 500 server errors.. Not sure what an be done to recover if anything.
What does a web program do if this occurs?
Run time = 99999999
I've had a program run thru 33 computer hours
|
|
|
Post by meerkat on Jan 3, 2013 11:34:19 GMT -5
From what I can read, it looks like your are running multiple task to get this information. Why not run a single task? It could be that there are too many task doing httpGet when it could be better to run it once..
Just my best guess.
|
|
|
Post by ezmoney on Jan 3, 2013 12:53:56 GMT -5
I am linking to different web addresses for data.
How do I run that to run it once? One get? mmm I guess I don't see how that can be done.. Clue me in.. This is just one program... Otherwise I could run each git with one program.. but that is very time consuming. and i can't type fast enought to create the link then post the program for 1 get and then run it.. That would be too labor intensive...
|
|
|
Post by meerkat on Jan 5, 2013 9:29:02 GMT -5
Here again I'm not sure what you are trying to do.
But assuming you are reading a bunch of web sites. It seemed like you had a lot of programs doing it. But using a singel program, I was suggesting you read them all up front. And because processing the huge html files takes time do it after.
I wrote a quickie program to show that process"
db$ = "webData.db" sqliteconnect #sql, db$
sql$ = "SELECT * FROM sqlite_master WHERE tbl_name = 'webInfo'" #sql execute(sql$) rows = #sql ROWCOUNT() 'Get the number of rows
if rows = 0 then sql$ = " CREATE TABLE webInfo ( webPage text, webDate text, webData text)" #sql execute(sql$) end if sql$ = "DELETE FROM webInfo" ' clear the old data #sql execute(sql$)
' --------------------------------- ' enter your list of sites to visit ' make this dim as big as you need ' -------------------------------- dim theWeb$(10) theWeb$(1) = "http://www.kneware.com" theWeb$(2) = "http://google.com" theWeb$(3) = "" theWeb$(4) = "" theWeb$(5) = "" theWeb$(6) = "" theWeb$(7) = "" theWeb$(8) = "" theWeb$(9) = "" theWeb$(10) = ""
on error goto [loop]
[loop] i = i + 1 web$ = trim$(theWeb$(i)) if web$ = "" then goto [getDone] print "Working on web Site:";web$ webData$ = httpGet$(web$) webData$ = dblQuote$(webData$) sql$ = "INSERT INTO webInfo VALUES('";web$;"','";date$()+" "+time$();"','";webData$;"')" #sql execute(sql$) goto [loop]
[getDone] print "==================== Gathering data complete ==================" sql$ = "SELECT * FROM webInfo ORDER BY webPage" #sql execute(sql$) WHILE #sql hasanswer() #row = #sql #nextrow() webPage$ = #row webPage$() webDate$ = #row webDate$() webData$ = #row webData$()
' ----------------------------------------------------------- ' Not sure what you are looking for . ' but this would be the place to gleen the data you want ' Process rendering data after gathering ' for max speed of getting data ' ------------------------------------------------------------ webData$ = strip$(webData$) ' strip junk webData$ = strRep$(webData$,chr$(9),"") ' get rid of tabs webData$ = tags$(webData$,"|") ' replace html tags with | webData$ = strRep$(webData$," |","|") ' get rid of space| to single |
print webPage$;chr$(9);webDate$ print "===========================================================" print webData$ print "===========================================================" WEND
end ' ----------------------------------------- ' strip junk ' ----------------------------------------- FUNCTION strip$(str$) strip$ = "" for i = 1 to len(str$) a$ = MID$(str$,i,1) a = ASC(a$) if a > 31 then if a < 127 then if a$ <> "'" then if a$ <> """" then strip$ = strip$ + a$ end if end if end if end if next i END FUNCTION
' ----------------------------------------- ' Convert single quotes to double quotes ' ----------------------------------------- FUNCTION dblQuote$(str$) i = 1 qq$ = "" while (word$(str$,i,"'")) <> "" dblQuote$ = dblQuote$;qq$;word$(str$,i,"'") qq$ = "''" i = i + 1 WEND END FUNCTION
' -------------------------------- ' string replace rep str with ' -------------------------------- FUNCTION strRep$(str$,rep$,with$) ln = len(rep$) ln1 = ln - 1 i = 1 while i <= len(str$) if mid$(str$,i,ln) = rep$ then strRep$ = strRep$ + with$ i = i + ln1 else strRep$ = strRep$ + mid$(str$,i,1) end if i = i + 1 WEND END FUNCTION
' -------------------------------- ' Replace HTML <tags> with$ ' -------------------------------- FUNCTION tags$(tag$,with$) i = instr(tag$,"<") j = instr(tag$,">",i) WHILE j > i ln = (j - i) + 1 x$ = mid$(tag$,i,ln) tag$ = left$(tag$,i -1) + with$ + mid$(tag$,j+1) i = instr(tag$,"<") j = instr(tag$,">",i)
WEND tags$ = tag$ END FUNCTION
|
|
|
Post by ezmoney on Jan 28, 2013 0:34:24 GMT -5
Well I think the time perameter is working well. I've seen 33 hours of computer time on the Windows Task Manager.. But just recently I saw 67 hours of computer time... So I think those time parms are set reasonable well..
Thanks for the info...
I am still plegued by the 500 internet service error.. It is a constant battle to keep restaring programs.
|
|
|
Post by ezmoney on Jan 28, 2013 0:43:03 GMT -5
That is a very nice program you wrote.
But I would need about 280,000 places for the input data and typing that out would be a big job.
Doubt if the program would survive such massive core storage requirements.
SQL might be nice but again I would just rather put the data in a file and I can work directly with it and not thru the web SQL.
Thanks for your input and seeing your very nice code.
|
|