Hi All,
We use TableFile to backup data to the USR drive, and also send it via FTPClient.
Problem is, some of the sites have intermittant 3G coverage so often the file cannot be sent. If it still can't be sent by the time the next file is created, it stays in USR drive unsent and we have to manually download it.
Anyone have a good method for resending multiple files?
My current method is to copy the file TableFile creates into another file prefixed with 'Temp'. Then every 6 hours try to FTP all the files with 'Temp' in them. If the transfer was successful they get deleted.
This works fine unless the USR drive is full, which means the copied file has 0 byte size.
Any ideas how I would go about determining which are the oldest files on the USR drive so I can delete them if there is no space?
Thanks
Ross
There is a FileTime function in the datalogger's instruction set, but it works only on files created with FileOpen (though it seems it might make sense to expand this, if possible). The only thing that would distinguish the age of a file would be the suffix (parsing that would be a headache).
Do you also perform data collection on this station using LoggerNet? If so, one option would be to set up file retrieval on a schedule. In LoggerNet 3.4.1, you can use LNCmd.exe running as a task to do this (see the PDF in C:\program files\campbellsci\LoggerNet). In LN 4.0, this functionality has been enhanced as an "Image Retrieval" tab for a device in the Setup screen. (In the 4.1 betas, it is now called "File Retrieval" since in reality you can retrieve any kind of file, not just an image).
One of these two options would allow you to set up a schedule where all USR:Temp*.dat files are retrieved and then deleted from the drive.
Regards,
Dana
There is a FileTime function in the datalogger's instruction set, but it works only on files created with FileOpen (though it seems it might make sense to expand this, if possible).
FYI -- Future operating systems will allow FileTime to work with other files (not just files created with FileOpen), if the FileHandle parameter [i.e., FileTime (FileHandle)] is a string.
Dana W.
Hi geometrikal,
We also do the same thing as you are ie. creat files on USR and send via ftp.
A nice simple way we do it is to send all files sitting on the USR drive each send interval. If you get a success on your FTPClient instruction then you delete all the files on the USR drive (using FileManage:8 instruction) otherwise you leave them there to be sent next time. This would mainly be applicible if you are not keeping too many files on the USR drive. We control the amount of files and the cyclic turnover of files on the USR with the Tablefile instruction. In my case I keep only 7 files (1 week of daily files). To send all files on the USR drive I 1st load up an array variable with all the filenames on USR. This can be done using the following code:
Function FilesOnUSR
FilesOnUSR = FileList ("USR",push_FileList())
ExitFunction
EndFunction
Dim fileNo
Public noOfFilesToPush
'get the number of files on the USR drive & load their names into a string array - push_FileList()
noOfFilesToPush = FilesOnUSR()
For fileNo = 1 To noOfFilesToPush
etc etc
Its not overly efficient but it works and as long as you are within your data allowance then no problems. We actually use cellular modems to connect to the internet with SIM account with 5Mb/month allowed for $5/month.
(removed)
* Last updated by: Sam on 1/15/2011 @ 10:35 PM *
Two common methods for managing the transfer of files from the logger using FTP, Email, SendFile, etc. include:
1) File name buffer - good when you want to keep the files even after a successful transfer - What if the file transferred became corrupted? If caught in time, you could manually retrieve the file again.
2) File deletion - good when you want to clear the drive of all files transferred - Easy to view drive contents and see if a backlog of files exists.
--------------------------------------------
Here is an example of (1) - keeping a buffer of file names that need to be transferred.
Public PTemp, batt_volt
'TABLE FILE DECLARATIONS
Public OutStat As Boolean
Public LastFileName As String * 31
'SERVER DECLARATIONS
Const Host = "ftp.server.com"
Const User = "user"
Const Pass = "pass"
Const RDir = "/"
Public LName As String * 31
Public RName As String * 31
Public Success As Boolean
'FILE QUEUE DECLARATIONS
Const QL = 10 'queue buffer size
Public Queue(QL) As String * 31
' SUB ROUTINE FOR ADDING TO QUEUE
Sub QueueAdd()
Dim I
If OutStat Then 'add new file to queue
For I = 1 To QL 'find an empty spot
If Queue(I) = "" Then
Queue(I) = LastFileName
ExitFor 'queued so exit
EndIf 'end if queue
Next I 'end for I
If I > QL Then 'there wasn't room, make some and add
Move (Queue(1),QL-1,Queue(2),QL-1) 'shift queued names down - 2 to 1, 3 to 2, 4 to 3, etc.
Queue(QL) = LastFileName
EndIf
EndIf 'end if outstat
EndSub
'SUB ROUTINE FOR REMOVING OLDEST ITEM IN QUEUE
Sub QueueRemove()
Move (Queue(1),QL-1,Queue(2),QL-1) 'shift queued names down - 2 to 1, 3 to 2, 4 to 3, etc.
Queue(QL) = "" 'clear trailing name
EndSub
'DATA TABLES
DataTable (Test,1,1000)
DataInterval (0,1,Sec,10)
TableFile ("USR:Test_",8,QL,30,0,Sec,OutStat,LastFileName)
Sample (1,batt_volt,FP2)
Sample (1,PTemp,FP2)
Call QueueAdd()
EndTable
DataTable(TxLog,1,1000)
Sample (1,Success,Boolean)
Sample (1,LName,String)
Sample (1,RName,String)
EndTable
BeginProg
SetStatus("USRDriveSize",1000000)
Scan (1,Sec,0,0)
PanelTemp (PTemp,250)
Battery (batt_volt)
CallTable Test
NextScan
SlowSequence
Do While TRUE
Do While (Queue(1) <> "") 'If queue is not empty
LName = Queue(1)
RName = Replace (Queue(1),"USR:","")
If FileTime(LName) > 0 Then 'if file exists
Success = FTPClient (Host,User,Pass,LName,RDir + RName,2) 'attempt to ftp
CallTable (TxLog) 'diagnostics table
Delay (1,4,Sec) 'pause after attempt, yes this is a good idea
If Success Then Call QueueRemove() Else ExitDo 'send succeeded or failed, if failed exit and try later
Else 'file does not exist
Call QueueRemove()
EndIf 'end if filetime
Loop 'end do while queue
Delay (1,10,Sec) 'Control SlowSequence Execution Frequency
Loop 'end do while true
EndSequence
EndProg
--------------------------------------------
Here is an example of (2) - Listing all the files on the drive and deleting them when your done with the transfer. Additionally, this example shows how a Sub routine can be created to sort files on a drive by file time stamps.
Public PTemp, batt_volt
Dim N
'TABLE FILE DECLARATIONS
Public OutStat As Boolean
Public LastFileName As String * 31
'SERVER DECLARATIONS
Const Host = "ftp.server.com"
Const User = "user"
Const Pass = "pass"
Const RDir = "/"
Public LName As String * 31
Public RName As String * 31
Public Success As Boolean
'FILE QUEUE DECLARATIONS
Public NFile
Const QL = 10 'queue buffer size
Public Queue(QL) As String * 31
Public FTime(QL) As Long
Const NL = 31 'this must match the length of the strings being passed into array
Const NF = 10
Sub ListFile(NumFiles, Names(NF) As String * NL, Times(NF) As Long,Drive As String, Order, Ext As String)
'INPUTS:
'*** DRIVE (string variable) - Drive that files should be listed for - "CPU", "USR", "CRD", "USB"
'*** ORDER (Float variable) - Order that files should be returned - 0=unordered, 1=ascending order by file time stamp
'*** EXT (string variable) - (not used,reserved) - File extension filter
'OUTPUTS:
'*** NUMFILES (float variable) - Number of files returned. -1= drive does not exist, -2=NAMES() not a variable
'*** NAMES() (string array) - List of file names
'*** TIMES() (long array) - Time stamps for list of files
' NOTES:
' Sorting alogorithm is "Selection sort", okay for small lists, inefficient for very large lists.
' Used b/c it does not require creation of temp/working list for sorting
Dim I, J
Dim swapJ
Dim SwapTime As Long
Dim SwapName As String * NL
'initialize
NumFiles = 0
Move (Names(),NF,"",1)
Move (Times(),NF,0,1)
'get list of files on drive
NumFiles = FileList (Drive,Names())
'if FileList returns < 1 then no files or encountered error
If NumFiles < 1 Then ExitSub
'Determine final list of file names and file times
For I = 1 To NumFiles
If Names(I) = "" Then ExitFor 'unexpected end of list
'insert filtering here when implemented
'will also need to update NumFiles if list is filtered
Times(I) = FileTime(Names(I))
Next I
If NumFiles > 1 Then
If Order = 1 Then 'ascending
'Sort TimeStamps and move FileNames accordingly
For I = 1 To NumFiles-1
SwapTime = Times(I)
SwapName = Names(I)
swapJ = I
For J = I+1 To NumFiles
If Times(J) < SwapTime Then
SwapTime = Times(J)
SwapName = Names(J)
swapJ = J
EndIf
Next J
Times(swapJ)=Times(I)
Times(I) = SwapTime
Names(swapJ) = Names(I)
Names(I) = SwapName
Next I
ElseIf Order = 2 Then 'decending
'Sort TimeStamps and move FileNames accordingly
For I = 1 To NumFiles-1
SwapTime = Times(I)
SwapName = Names(I)
swapJ = I
For J = I+1 To NumFiles
If Times(J) > SwapTime Then
SwapTime = Times(J)
SwapName = Names(J)
swapJ = J
EndIf
Next J
Times(swapJ)=Times(I)
Times(I) = SwapTime
Names(swapJ) = Names(I)
Names(I) = SwapName
Next I
EndIf
EndIf
EndSub
'DATA TABLES
DataTable (Test,1,1000)
DataInterval (0,1,Sec,10)
TableFile ("USR:Test_",8,QL,0,30,Sec,OutStat,LastFileName)
Sample (1,batt_volt,FP2)
Sample (1,PTemp,FP2)
EndTable
DataTable(TxLog,1,1000)
Sample (1,Success,Boolean)
Sample (1,LName,String)
Sample (1,RName,String)
EndTable
'MAIN PROGRAM
BeginProg
SetStatus("USRDriveSize",1000000)
Scan (1,Sec,0,0)
PanelTemp (PTemp,250)
Battery (batt_volt)
CallTable Test
NextScan
SlowSequence
Do While TRUE
Call ListFile(NFile,Queue, FTime,"USR",1,"")
If NFile > 0 Then
For N = 1 To NFile
LName = Queue(N)
RName = RDir + Replace (Queue(N),"USR:","")
Success = FALSE
Success = FTPClient (Host,User,Pass,LName,RName,2) 'attempt to ftp
CallTable (TxLog) 'diagnostics table
Delay (1,4,Sec) 'pause after attempt, yes this is a good idea
If Success Then
FileManage (Queue(N),8 ) 'remove file from drive
Else
ExitFor 'exit and try later
EndIf
Next N
EndIf
Delay (1,10,Sec) 'Control SlowSequence Execution Frequency
Loop 'end do while true
EndSequence
EndProg
* Last updated by: Sam on 1/15/2011 @ 10:36 PM *
Sam
How could i filter with extension?
Im trying to make a sorted list with only a kind of extension (csv) , and when i call the sub with extension it keeps making the list with the cr6 datalogger program , and send it to my sftp , then it tries to delete it.
That's because im working with only a CPU memory.
I need only to make an array with csv files and then only delete that extension of files , not other.