BSBI Distribution Database > message board

Sync files slow to receive DDb acknowledgement

Please report any problems with the website. Suggestions for changes are also very welcome.

Sync files slow to receive DDb acknowledgement

by AndyAmphlett » Sun Sep 11, 2016 5:38 am

Tom,

the new system whereby sync files are imported to the DDb within 15 - 20 minutes of sending, has been working perfectly. I have quickly adopted a workflow, submitting records as soon as they are entered, and then validating them as soon as I get the DDb acknowledgement.

However something has changed over the last day or so. My last two sync files did not receive an automated DDb acknowledgement for several hours, instead of a few minutes. For example last night I sent a sync (with just 13 records made that day). I sent the sync at 21.36hrs, and had the automated reply saying received at by 'Bob Ellis' at 21.39. The DDb acknowledgement arrived at 03.34 the next morning, c. 6 hours later. I don't know if the delay is the DDb sending the acknowledgement, or if record import has slowed.

Andy.
AndyAmphlett
 
Posts: 408
Joined: Fri Nov 23, 2012 9:32 am
name: Andy Amphlett

Re: Sync files slow to receive DDb acknowledgement

by admin » Sun Sep 11, 2016 10:34 am

Hi Andy,

Sorry about the slow imports. There have been database problems over the past couple of days which had caused scheduled DDb tasks to somewhat clog up. The database is split between slow and fast storage space on the server and we'd begun to run out of space on the 'fast' partition. To allow that problem to be resolved I had to temporarily move some of the heavily used tables to the slow partition. It then became difficult to put the moved tables back again, because scheduled tasks were over-running so that there was never a point when the system was not under heavy load.

Things should be back to normal now and the backlog of sync files is clear.

More generally there are occasions when synch files will be slower than usual to import, because the system can run only one import task at a time. Synch files will always be pulled from the hub quickly (on a 15 min cycle) - it is that which triggers the first email notification that appears to come from Bob, but the files are then queued if another import task is already underway.

The most recent BSBI e-newsletter includes a request that all VCRs reset their synch record with the hub and send a new synch file - that may result in a series of large files to import over the next few days.

Tom
Tom Humphrey
Database Officer, Botanical Society of Britain and Ireland (BSBI)
c/o Centre for Ecology and Hydrology,Maclean Building, Crowmarsh Gifford, Wallingford, Oxon, OX10 8BB, UK.
tom.humphrey@bsbi.org
User avatar
admin
 
Posts: 438
Joined: Tue Nov 20, 2012 4:16 pm
name: Tom Humphrey

Re: Sync files slow to receive DDb acknowledgement

by admin » Mon Sep 12, 2016 11:08 am

Synch file import may be slow again today (Monday 12th) as I've just started a large (million record) import which could take all day to run and will block MapMate imports until it is finished. Synch files will still be pulled from the hub and queued, so there is no problem with sending them.
Tom Humphrey
Database Officer, Botanical Society of Britain and Ireland (BSBI)
c/o Centre for Ecology and Hydrology,Maclean Building, Crowmarsh Gifford, Wallingford, Oxon, OX10 8BB, UK.
tom.humphrey@bsbi.org
User avatar
admin
 
Posts: 438
Joined: Tue Nov 20, 2012 4:16 pm
name: Tom Humphrey

Re: Sync files slow to receive DDb acknowledgement

by AndyAmphlett » Mon Sep 12, 2016 10:01 pm

Tom,

is it ok to send a sync file to the Hub before receipt of an email saying that the earlier sync file had been imported? In the old system, a second sync would overwrite the first. It makes a nice workflow to just enter a batch or records and send a sync straight away.

Thanks,

Andy.
AndyAmphlett
 
Posts: 408
Joined: Fri Nov 23, 2012 9:32 am
name: Andy Amphlett

Re: Sync files slow to receive DDb acknowledgement

by admin » Mon Sep 12, 2016 10:18 pm

Hi Andy,

During the import process the system sends two emails, the first sent via MapMate (ostensibly from Bob Ellis), indicates that the file has been pulled from the hub. Once you've received that then it's safe to send a new file to the hub.

Unfortunately the import delays today will be repeated tomorrow as the large import failed and will need to be retried.

Tom
Tom Humphrey
Database Officer, Botanical Society of Britain and Ireland (BSBI)
c/o Centre for Ecology and Hydrology,Maclean Building, Crowmarsh Gifford, Wallingford, Oxon, OX10 8BB, UK.
tom.humphrey@bsbi.org
User avatar
admin
 
Posts: 438
Joined: Tue Nov 20, 2012 4:16 pm
name: Tom Humphrey


Return to Bugs and suggestions

cron