db maintenance, file processing off

[bobzilla]

wanted to let folks know that file processing will be off for much of this weekend for some db maintenance. feel free to keep uploading files, they will all get processed afterwards. we apologize for the inconvenience.

[whitedice]

wanted to let folks know that file processing will be off for much of this weekend for some db maintenance. feel free to keep uploading files, they will all get processed afterwards. we apologize for the inconvenience.

Hey just curious, but what about the Web Maps? I see the file processing was turned back on yesterday, but the Web Maps are still down?

[bobzilla]

thanks, i'm spinning that up now

[i_do_dew]

Looks like the world and Europe maps need to be re-genned as well, or maybe just their front page thumbs.

Of course, like anything so much to do and only so many hampsters. I would suggest rabbits but it is harder getting useful work out of those because all they do is re-produce.

[uhtu]

rabbits eh. i'll pass it on to the interns!
they're being directed to re-create those maps just as soon as the mapserver has cleared the planet.

[AZ_WD'er]

Looks like the world and Europe maps need to be re-genned as well, or maybe just their front page thumbs.

Of course, like anything so much to do and only so many hampsters. I would suggest rabbits but it is harder getting useful work out of those because all they do is re-produce.

Looks like the rabbits got REAL busy and reproduced a bit too fast. 800+ files in que and none processed in at least 20+ hours. Must be Party Time Rabbit Style.

[uhtu]

on the contrary, they have been processing pretty much continuously during that time (and are still going); the
dark magic that updates the processed/remaining dosen't get tweaked as often as would be handy to assuage concerns.

[dagaroth]

lol, whoops, I should have read before I uploaded, I just dumped about 400 files up there too

Sorry to anyone that gets in behind me

[dagaroth]

actually, processing seems slow, it's only moved about 60-70 files in 5 hours

[Blaze312]

Dammit...

I got stuck behind you.

I am queued 440 now.

Just a question dagaroth, why so many files?

I think im beginning to finally understand this:
14,759,557 points from 895,974,862 unique observations.
I always wondered why there was almost a billion observations and less then 15 million actual points.

If i did my math correctly we are averaging 1 new point every 60 files upload. Why would this be? I mean i've often uploaded the same points as part of a stumble upload, but i almost always get at least 10 new points.

Sorry to go on on a rant... Just curious....

[uhtu]

networks are (network)

observations are (network,lat,lon)

so you will always have at least as many observations as networks, and usually many many more.

[whitedice]

Each file has multiple locations for 1 point. The *.gps file from kismet is exactly what it is... a file filled with gps locations continuously as you are moving, and recording what is at each point. I'm roughly guessing at that.

I have uploaded files in the past that have taken 70+ minutes to individually calculate. Imagine driving for 8 hours, and recording 13,000 points. Out of the 13,000, getting only 2,000 new. Lots of database searching, and trangulating. It wouldn't surprise me that the bigger the database gets, it will just take longer in general.

I usually only have about 20-30 files for a weeks worth of collecting while I'm driving

[dagaroth]

actually, I went back later and looked, it was 227.

I wardrive every day. If I'm in the car, my rig is on. Usually it's just to/from work, or to/from my kids school or daycamp, so most of my logs are less than 20 miles.

On actual wardrives, I plan ahead using google maps and google earth to maximize my routes, and I plan in stop points to shut down kismet and start a new logfile. I have a problem with my setup where if I run for more than an hour or so, everything locks, and I dont log anything, resetting before I hit an hour seems to fix that. I think it's my rig, I just need time/money to build a new one.

I'd also gotten lazy and didnt dump my stuff onto my USB stick in over a month

8 hour wardrives? damn man, are you like a courier or something? I cant actually wardrive for more than an hour or so without getting bored With the exception of the long family trips we take, or my occasional trip to pick up something I got off ebay, most of my planned wardrives are around an hour

[dagaroth]

and I just realized you're in Cinci whitedice.. are you the reason why the points I got in cleveland and medina where so low last summer

[whitedice]

8 hour wardrives? damn man, are you like a courier or something? I cant actually wardrive for more than an hour or so without getting bored With the exception of the long family trips we take, or my occasional trip to pick up something I got off ebay, most of my planned wardrives are around an hour

My job until today (i just quit to go back to school) was driving a semi around the Eastern US. I've logged 128,000 miles in 15 months.

To give you an idea here is one file, in my uploads list.
4344 seconds (72.4 minutes)
Whitedice-Jul-09-2008-1.csv
2,465,968 bytes
8,568 File lines
1,002 new networks w/ gps
1,003 total new discovered networks
8,567 total networks in file
8,567 total new locations

That particular file, I drove around Raleigh, NC picking up a load that went down near Charleston, SC. I turn on kismet, and did my job, and when I stop to sleep, I turned it off.

Someone has done a bunch to the east of cleveland out near 271. I drove downtown, and some major roads and universities around the area, and most of the stuff to the west is me (between Sandusky and Cleveland). It isn't all me, but a lot of it is. I did that before I started driving a truck with my SUV setup.

I'm in the process of creating an animated gif of my weekly additions while working. Kind of neat to see... In my opinion.