How to run an endless session

[ezmoney]

I do a lot of batch work and then assembled the data later.
I want to automate this and get it to spawn new task as the machine load declines due to
task being completed.
My question is, .. is there a way to automate the process?

I am getting a lot of errors saying expired session even some very short notice ones.

Is there a way to recover from these and resume processing at some point?
I have the error code trap built in but that don't seem to catch some errors and then
the program is dead. Now when that happens I must restart at a point in the program or
get the same data twice.. or more times for each restart.

Suggestion to Carl for your consideration.
I was wondering if maybe Carl would build into his system a way to call programs and get past the logon some how. And to allow an input data file to fill in the answers rather than
a keyboard like redirected input with windoes. This would greatly amplify the power of the code. Of course starting programs is nice but also there need to eventaully end or all your core is taken up, so there needs to be a way to kill the console and the program itself.

[meerkat]

I guess I'm not exactly sure what you are trying to do.
But I'll give it my best guess.

First - about timeouts.
There are 2 timeouts in Run Basic, Session and Run time.
- The session is the amount of time you will wait for a person to respond. For example if you have a session timeout of 2 hours, and your program ask a person a question, they have two hours to respond.
- Run time is the time a program can run. This is basically to prevent loops. If you set the run time to about 30 seconds and your program continually runs for that length of time without a wait for user input it will time out.

Go the to Run Basic [Preferences] tab to set the preferences..

I assume you have a web page that your users use to get into your system. Lets assume you have a project called test. You web page probably has a link that looks something like this.
<A href="http://localhost:8008/seaside/go/runbasicpersonal?app=test" target="_blank"></A>
Unless you use the RunBasic [Publish] tab and mark test as published it will ask for a login.

You can use the RUN command to run other tasks..

Not sure I answered your questions. But hope this helped.
Dan

[ezmoney]

I work from my computer to get data from the web.
I have put up about 30 programs but all came back with expired session in about 1 hour.
My profile is set much much higher in the 9999 run time.

I started over and some of those posted quit with expired session already and they haven't been up very long. The rest are still running... Mmmm something strange is going on here and
I don't know what it is or how to solve it.. Can anybody clue me in?

[StefanPendl]

The preferences tab is described in the quick start guide.

The runtime limit is the amount of time a program can run without user intervention.
The session limit is the amount of time a program can run at all.

It seems that your session limit is still at 1 hour (= 3600 seconds), so you have to increase that too.

[meerkat]

It's hard to know what you are trying to do without seeing some code.

However you could set a timeout in your web page and have it run itself when it times out.

This is set to timeout in about 28 Hours.

<script type="text/JavaScript">
<!--
setTimeout("location.href = 'http://localhost:8008/seaside/go/runbasicpersonal?app=yourApp';",99999);
-->
</script>


[ezmoney]

Basically I do the
open #1, git input file
Open #4 for output
limit=1000
[loop]
read the git$

ss$=httpget$(git$)

data$=function(ss$)

store the result

print #4,data$
rec=rec+1
if rec < limit then [again]
close #4
close #1
end

I am getting a lot of 500 server errors.. Not sure what an be done to recover if anything.

What does a web program do if this occurs?

Run time = 99999999

I've had a program run thru 33 computer hours

[meerkat]

From what I can read, it looks like your are running multiple task to get this information.
Why not run a single task?
It could be that there are too many task doing httpGet when it could be better to run it once..

Just my best guess.

[ezmoney]

I am linking to different web addresses for data.

How do I run that to run it once?
One get? mmm I guess I don't see how that can be done.. Clue me in..
This is just one program... Otherwise I could run each git with one program..
but that is very time consuming. and i can't type fast enought to create the link then post
the program for 1 get and then run it..
That would be too labor intensive...

[meerkat]

Here again I'm not sure what you are trying to do.

But assuming you are reading a bunch of web sites. It seemed like you had a lot of programs doing it. But using a singel program, I was suggesting you read them all up front. And because processing the huge html files takes time do it after.

I wrote a quickie program to show that process"

db$ = "webData.db"
sqliteconnect #sql, db$

sql$ = "SELECT * FROM sqlite_master WHERE tbl_name = 'webInfo'"
#sql execute(sql$)
rows = #sql ROWCOUNT() 'Get the number of rows

if rows = 0 then
 sql$ = "
 CREATE TABLE webInfo (
 webPage text,
 webDate text,
 webData text)"
 #sql execute(sql$)
end if
sql$ = "DELETE FROM webInfo" ' clear the old data
#sql execute(sql$)

' ---------------------------------
' enter your list of sites to visit
' make this dim as big as you need
' --------------------------------
dim theWeb$(10)
theWeb$(1) = "http://www.kneware.com"
theWeb$(2) = "http://google.com"
theWeb$(3) = ""
theWeb$(4) = ""
theWeb$(5) = ""
theWeb$(6) = ""
theWeb$(7) = ""
theWeb$(8) = ""
theWeb$(9) = ""
theWeb$(10) = ""

on error goto [loop]

[loop]
i = i + 1
web$ = trim$(theWeb$(i))
if web$ = "" then goto [getDone]
print "Working on web Site:";web$
webData$ = httpGet$(web$)
webData$ = dblQuote$(webData$)
sql$ = "INSERT INTO webInfo VALUES('";web$;"','";date$()+" "+time$();"','";webData$;"')"
#sql execute(sql$)
goto [loop]

[getDone]
print "==================== Gathering data complete =================="
sql$ = "SELECT * FROM webInfo ORDER BY webPage"
#sql execute(sql$)
WHILE  #sql hasanswer()
#row = #sql #nextrow()
webPage$ = #row webPage$()
webDate$ = #row webDate$()
webData$ = #row webData$()

' -----------------------------------------------------------
' Not sure what you are looking for .
'  but this would be the place to gleen the data you want
' Process rendering data after gathering
'  for max speed of getting data
' ------------------------------------------------------------
webData$ = strip$(webData$) ' strip junk
webData$ = strRep$(webData$,chr$(9),"") ' get rid of tabs
webData$ = tags$(webData$,"|")          ' replace html tags with |
webData$ = strRep$(webData$," |","|")   ' get rid of space| to single |

print webPage$;chr$(9);webDate$
print "==========================================================="
print webData$
print "==========================================================="
WEND

end
' -----------------------------------------
' strip junk
' -----------------------------------------
FUNCTION strip$(str$)
strip$ = ""
for i = 1 to len(str$)
a$ = MID$(str$,i,1)
a = ASC(a$)
if a > 31 then
 if a <  127 then
  if a$ <> "'" then
   if a$ <> """" then
    strip$ = strip$ + a$
   end if
  end if
 end if
end if
next i
END FUNCTION

' -----------------------------------------
' Convert single quotes to double quotes
' -----------------------------------------
FUNCTION dblQuote$(str$)
i    = 1
qq$  = ""
while (word$(str$,i,"'")) <> ""
dblQuote$   = dblQuote$;qq$;word$(str$,i,"'")
qq$ = "''"
i   = i + 1
WEND
END FUNCTION

' --------------------------------
' string replace rep str with
' --------------------------------
FUNCTION strRep$(str$,rep$,with$)
ln  = len(rep$)
ln1 = ln - 1
i = 1
while i <= len(str$)
   if mid$(str$,i,ln) = rep$ then
       strRep$ = strRep$ + with$
       i = i + ln1
   else
       strRep$ = strRep$ + mid$(str$,i,1)
   end if
i = i + 1
WEND
END FUNCTION

' --------------------------------
' Replace HTML <tags> with$
' --------------------------------
FUNCTION tags$(tag$,with$)
i = instr(tag$,"<")
j = instr(tag$,">",i)
WHILE j > i
ln = (j - i) + 1
x$ = mid$(tag$,i,ln)
tag$ = left$(tag$,i -1) + with$ + mid$(tag$,j+1)
i = instr(tag$,"<")
j = instr(tag$,">",i)

WEND
tags$ = tag$
END FUNCTION


[ezmoney]

Well I think the time perameter is working well.
I've seen 33 hours of computer time on the Windows Task Manager..
But just recently I saw 67 hours of computer time...
So I think those time parms are set reasonable well..

Thanks for the info...

I am still plegued by the 500 internet service error..
It is a constant battle to keep restaring programs.

[ezmoney]

That is a very nice program you wrote.

But I would need about 280,000 places for the input data and typing that out would be a big job.

Doubt if the program would survive such massive core storage requirements.

SQL might be nice but again I would just rather put the data in a file and I can work directly with it and not thru the web SQL.

Thanks for your input and seeing your very nice code.