Page 1 of 1
Upload queue not moving
Posted: Fri Nov 10, 2006 1:42 pm
by teststrips
I could just be expecting too much from the upload system, but I uploaded a file over an hour ago, and it didn't move at all... still in queued spot #2 - is this normal?
Posted: Fri Nov 10, 2006 2:27 pm
by teststrips
well, it moved now + has completed
Looking at the maps they don't show up though... how long does that take?
Posted: Fri Nov 10, 2006 2:37 pm
by teststrips
Well I found an answer to my own question, but will post here for future refrence of other newbies - uploads into the system don't get processed for use in the mapping system until somewhere between 1-9AM CST
If you want to see your results realtime Jigle/Digle appear to be realtime viewers - download the application - then download web maps
Someone correct me if I'm wrong on anything
well...
Posted: Fri Nov 10, 2006 11:56 pm
by uhtu
extremely large files with lots of redundant data take a while to process, and will cause delays in the queue processing.
the maps are updated once per day, or whenever the mapserver cluster admins reboot it.
Re: well...
Posted: Sat Nov 11, 2006 12:14 am
by Dutch
... redundant data ...
Now there's an oxymoron wiglewise. None of the data we submit, is redundant! *waves pokingstick^sm at Uhtu*
Dutch
we are legion
Posted: Sun Nov 12, 2006 1:49 am
by uhtu
we add all posted data to the system, but a series of identical posts are coalesced, and just eat time before they are discarded: posting the same file twice just wastes heat and time :-)
Re: well...
Posted: Sun Nov 12, 2006 1:54 am
by i_do_dew
... redundant data ...
Now there's an oxymoron wiglewise. None of the data we submit, is redundant! *waves pokingstick^sm at Uhtu*
Dutch
Different data for the same host isn't redundant, duplicate data is.
Re: we are legion
Posted: Sun Nov 12, 2006 2:43 am
by Dutch
we add all posted data to the system, but a series of identical posts are coalesced, and just eat time before they are discarded: posting the same file twice just wastes heat and time
You mean some people post the exact same file several times, thinking that it will skew the QOS indicator ? Maybe it's time to implement hashing and checksumming on submitted files to prevent such tarded behaviour ?
Dutch
sigh
Posted: Sun Nov 12, 2006 7:42 pm
by uhtu
it already dosen't do anything except waste time.
in general it seems to be folks who post aggregates of their drives instead of just their new data. so take an 11ty billion network file, go for a drive, see two nets, post all 11ty. the server already ignores everything except those two new ones, it just has to chew through everything to get to 'em.
Re: sigh
Posted: Sun Nov 12, 2006 8:07 pm
by Dutch
it already dosen't do anything except waste time.
in general it seems to be folks who post aggregates of their drives instead of just their new data. so take an 11ty billion network file, go for a drive, see two nets, post all 11ty. the server already ignores everything except those two new ones, it just has to chew through everything to get to 'em.
Ahh.. Thats retarded, all right.
I do submit my
individual stumble files, since I seem to recall that updated loggings will be used in the lat/long approximation calculations. But since I also keep an log of new, not previously detected networks, I could submit just those, in order to keep the processing time down on my 50+ Mb kismet gps logfiles.
Would you prefer the present upload of the individual stumble files, or the latter ?
Dutch
as with many things
Posted: Sun Nov 12, 2006 9:56 pm
by uhtu
smaller = faster. newer = better.
Re: as with many things
Posted: Sun Nov 12, 2006 11:57 pm
by Dutch
smaller = faster. newer = better.
So you're saying that I'm damned if I do, and damned if I don't ?
Actually, I think I will try and see how much time it saves, if I strip out the track log data from the kismet gps file, so it only contains actual network location data, before uploading it. I reckon that the wigle processing strips that out in the parsing anyway, so just saving some cpu cycles on the C-64 box that runs the Wigle DB
Dutch
Re: as with many things
Posted: Tue Nov 14, 2006 3:27 am
by pejacoby
I reckon that the wigle processing strips that out in the parsing anyway, so just saving some cpu cycles on the C-64 box that runs the Wigle DB
Hot damn, the DB box is a C-64 now? Awesome, I
thought things felt faster.
What's happening with the old Vic-20 now? Just processing Golomb Rules for distributed.net I suppose....